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Abstract 



Towards Hybrid Intensional Programming with JLucid, Objective 
Lucid, and General Imperative Compiler Framework in the GIPSY 

Serguei A. Mokhov 

Pure Lucid programs are concurrent with very fine granularity. Sequential Threads 
(STs) are functions introduced to enlarge the grain size; they are passed from server 
to workers by Communication Procedures (CPs) in the General Intensional Program- 
ming System (GIPSY). A JLucid program combines Java code for the STs with Lucid 
code for parallel control. Thus first, in this thesis, we describe the way in which the 
new JLucid compiler generates STs and CPs. JLucid also introduces array support. 

Further exploration goes through the additional transformations that the Lucid 
family of languages has undergone to enable the use of Java objects and their mem- 
bers, in the Generic Intensional Programming Language (GIPL), and Indexical Lucid: 
first, in the form of JLucid allowing the use of pseudo-objects, and then through the 
specifically-designed the Objective Lucid language. The syntax and semantic def- 
initions of Objective Lucid and the meaning of Java objects within an intensional 
program arc provided with discussions and examples. 

Finally, there arc many useful scientific and utility routines written in many im- 
perative programming languages other than Java, for example in C, C-|--|-, Fortran, 
Perl, etc. Therefore, it is wise to provide a framework to facilitate inclusion of these 
languages into the GIPSY and their use by Lucid programs. A General Imperative 
Compiler Framework and its concrete implementation is proposed to address this 
issue. 
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Chapter 1 
Introduction 



1.1 Thesis Statement 

In the previous prototype of the General Intensional Programming System (GIPSY) 
there existed limitations to its potential in distributed computing - lack of sequential 
threads and communication procedures. Additionally, the capabilities of Indexical 
Lucid and GIPL, the primary GIPSY's languages, were limited to only computing 
aspects without input/output, arrays, and some other essential features (e.g. math, 
non-determinism, dynamic loading) that exist in imperative (e.g. Java) languages. 
We discuss an extension to Generic Intensional Programming Language (GIPL) and 
Indexical Lucid with embedded Java - JLucid. A few problems are solved as an 
example using the enhanced language. 

JLucid brings embedded Java and most of its powers into Indexical Lucid in the 
GIPSY by allowing intensional languages to manipulate Java methods as first class 
value^ However, it is very natural to have objects with Java and manipulate their 
members in scientific intensional computation, yet JLucid fails to support that Java's 
capability. Hence, we design Objective Lucid to address this deficiency. We define the 
operational semantics of Objective Lucid, and give some examples of its application. 

Existence of JLucid, Objective Lucid, and GLU as well as many useful libraries 
written in other imperative languages, such as C/C++, Perl, Python, Fortran etc. 
demanded ability to use code written in those languages by intensional programs, 

^The Java methods are not referred to as "functions" as in functional programming - the Java 
methods can be passed around as values inside the Lucid part, but not to or from Java part of a 
GIPSY program. 
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naturally. Thus, we design a first version of the General Imperative Compiler Frame- 
work (GICF) as a part of the GIPSY to allow GIPSY programs to use virtually any 
combination of intensional and imperative languages at the meta level. This is a very 
ambitious goal; therefore, the proposal is the first iteration of the framework open for 
later refinements as it matures along with the corresponding changes to the run-time 
system. 

1.2 Contributions 

Primary contributions of this thesis are outlined below: 

• JLucid 

— Semantics of pseudo-free Java methods in Lucid programs 

— Design and implementation of JLucid and its compiler in the GIPSY 

• Objective Lucid 

— Semantics of the integration of Java objects in Lucid programs 

— Design and implementation of the Objective Lucid compiler 

• General Imperative Compiler Framework 

— Design and Implementation of the GICF 

— Embedding of a Java compiler in the GICF 

• WebEditor to edit, compile, and run GIPSY programs onhne 

• System Architecture Issues 

— Rework and refactoring of most existing system design, both at the archi- 
tectural and detailed design levels 

— Major rework of the architecture and detailed design of GIPC 

— Java sequential threads generation 

— Threaded and RMI communication procedures generation 
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— GIPSY Type SysteiiQ 

— GIPSY Exceptions Framework 

— Regression Testing Infrastructure 

— Unit Testing Automation with JUnit 

The last contributed items touch the rest of the GIPSY, the components and 
modules done by other team members. The integration performed (outside of the 
main scope of this thesis) demanded extensive testing. Without the integration and 
testing work, these other contributions wouldn't be possible. This also includes de- 
veloping and enforcing Coding Conventions and setting up project's CVS repository 
[MokOSbl IMok03al IMokOSb] for the entire project as this work is to become a manual 



for the current and future GIPSY developers and researchers. 



1.3 Scope of the Thesis 

While the Contributions section outlines the major work done, the below explains 
what was not done or exhibits some limitations at the time of this writing: 

• Integrated imperative compilers aren't native to the GIPSY, instead we call 
external compilers, such as javac, gcc, g++, nmake.exe, bc.exe, perl, etc. 
depending on a platform. 

• Even though the mechanism was designed and implemented to generate CPs and 
STs, only two of the concrete implementations of the actual CPs were done: for 
local execution and distributed execution by extending the RMI implementation 
done by Bo Lu. The other implementations of CPs for Jini, DCOM+, CORBA 
and others are being worked on by other team members at the time of this 
writing. 

• Semantic rules to have Java objects in Objective Lucid have been developed, 
but have not been formally proven to be correct. 

^Though the type system may seem not to be related to the architecture, but it impacted the 
design most of the main modules in it, so it was classified as architectural. 
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• When presenting GICF and the Preprocessor syntax, no semantic rules are 
given for any of parts of the hybrid programs, except for JLucid and Objective 
Lucid, i.e. the semantics of integrated Java itself or C constructs, etc. 

• JLucid and Objective Lucid are still in their experimental stage of development 
and it will take some time before they mature. 

1.4 Structure of the Thesis 

The next chapter provides the necessary background on the Lucid family of languages, 
its history, operational semantics, compiler frameworks, and hybrid programming. 
Then, it gives the context of this thesis, the GIPSY system, and the tools and tech- 
niques employed to make the contributions possible. The core of this thesis is based 
on three publications, namely |MPG05i IMPOSbl IMPOSaj . Chapter [3] describes the 
approach and methodology used to overcome and provide a solution to the prob- 
lems stated in Section [LT| Then, the design implementation details are presented in 
Chapter |4| Chapter [5] introduces the Regression Testing Suite for GIPSY and what 
kinds of tests were performed and their limitations. Finally, Chapter |6] and Chapter [7] 
conclude on the work done, discuss the results and limitations of the implementation, 
and lay down some paths towards enhancing the GIPSY in various areas further. At 
the end, there is a list of references. Bibliography, and an Appendix with most com- 
mon abbreviations found in this work, CP and ST interfaces, JLucid and Objective 
Lucid grammar generation scripts, etc., followed by an overall index. 
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Chapter 2 
Background 



While there is a complete and comprehensive set of references in the Bibliogra- 
phy chapter that was a great deal of help to the creation of this work, there are 
some keynotes that require special mention. The following are some of the re- 
lated readings that were sources of inspiration and invaluable informational food 
for thought. These include Joey Paquet's PhD thesis "Scientific Intensional Pro- 
gramming" |Paq99| , related hybrid intensional-imperative programming in various 
GLU-related work, such as |JD96t IJDA97j , other recent hybrid programming papers, 
such as |PKn4l IMSnil[SMn2] . the PhD thesis of Paula Bo Lu jHIni] and other theses 
of the GIPSY group, such as |Ren02l IDin04| ITao04| IWu02j , and semantics of program- 
ming languages in |Gro02ai IHJ02t IMoe04j . Additionally, since this work also deals 
with compiler frameworks, a general overview of existing frameworks is presented. 
An on-line encyclopedia, Wikipedia |WSoafaotw05] . was a valuable resource for the 
background and literature review, some of which is summarized in the sections that 
follow. 

2.1 Intensional Programming 

Intensional programming is a generalization of unidimensional contextual (also known 
as modal logic |Car47t IKri59t IKri63j ) programming such as temporal programming, 
but where the context is multidimensional and implicit rather than unidimensional 
and explicit. Intensional programming is also called multidimensional programming 
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because the expressions involved are allowed to vary in an arbitrary number of di- 
mensions, the context of evaluation is thus a multidimensional context. For example, 
in intensional programming, one can very naturally represent complex physical phe- 
nomena such as plasma physics (e.g. in Tensor Lucid in |Paq99| ), which are in fact 
a set of charged particles placed in a space-time continuum that behaves according 
to a limited set of laws of intensional nature. This space-time continuum becomes 
the different dimensions of the context of evaluation, and the laws are expressed nat- 
urally using intensional definitions |Paq99| . Joey Paquet's PhD thesis discusses the 
syntax and semantics of the Lucid language, designs GIPL and Tensor Lucid. While 
we omit the Tensor Lucid part, the reader is reminded about the basic properties of 
the Indexical Lucid and GIPL languages in the follow up sections in greater detail to 
provide the necessary context for the follow up work in Chapter [3] and Chapter |4} 

Intensional Logic 

Intensional programming (IP) is based on intensional (or multidimensional or modal) 
logic (where semantics was applied first by |Car47l lKri59t lKri63j ) . which, in turn, are 
based on Natural Language Understanding (aspects, such as, time, belief, situation, 
and direction are considered). IP brings in dimensions and context to programs 
(e.g. space and time in physics or chemistry). Intensional logic adds dimensions 
to logical expressions; thus, a non-intensional logic can be seen as a constant or a 
snapshot in all possible dimensions. Intensions are dimensions at which a certain 
statement is true or false (or has some other than a Boolean value). Intensional 
operators are operators that allow us to navigate within these dimensions. 

Temporal Intensional Logic 

Temporal intensional logic is an extension of temporal logic that allows to specify the 
time in the future or in the past. 

(1) El := it is raining here today 
Context: {place: here, time: today} 

(2) E2 := it was raining here 6e/ore(today) = yesterday 

(3) E3 := it is going to rain ai(altitude here + 500 m) after{todsLy) = tomorrow 
Let's take Ei from (1) above. Then let us fix here to Montreal and assume it is 
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a constant. In the month of March, 2004, with granularity of day, for every day, we 
can evaluate Ei to either true or false: 

Tags : 123456789... 
Values : FFTTTFFFT ... 

If you start varying the here dimension (which could even be broken down into 
X, Y, Z), you get a two-dimensional evaluation of Ei. 

City /Day 123456789... 

Montreal FFTTTFFFT ... 

Quebec FFFFTTTFF ... 

Ottawa FTTTTTFFF ... 

The purpose of this example is to remind the reader the basic ideas behind in- 
tensions and intensional programming and what dimensionality is by using natural 
language. What follows is formalization of the above in terms of the Lucid program- 
ming language. 



2.2 The Lucid Programming Language 

Let us begin by introducing the Lucid language history and which features of it came 
at different stages of its evolution to its present form. This is the necessary step to 
further illustrate the purpose of this thesis. 

2.2.1 Brief History and The Family 

From 1974 to Lucid Today: 

1. Lucid as a Pipelined Dataflow Language through 1974-1977. Lucid was intro- 
duced by Anchroft and Wadge in |AW76t IAW77] . Features: 

• A purely declarative language for natural expression of iterative algo- 
rithms. 

• Goals: semantics and verification of correctness of programming languages 
(for details see |AW76l EW77] l 
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• Operators as pipelined streams: one for initial element, and then all for 
the successor ones. 

2. Intensions, Indexical Lucid, GRanular Lucid (GLU, |JD96l[TDA97j ). circa 1996. 
More details on these two dialects are provided further in the chapter as they 
directly relate to the theme of this thesis. Features: 

• Random access to streams in Indexical Lucid. 

• First working hybrid intensional- imperative paradigm (C/Fortran and In- 
dexical Lucid) in the form of GLU. 

• Eduction or demand-driven execution (in GLU). 

3. Partial Lucid, Tensor Lucid, 1999 |Paq99| . 

• Partial Lucid is an intermediate experimental language used for demon- 
strative purposes in presenting the semantics of Lucid in |Paq99] . 

• Tensor Lucid dialect was developed by Joey Paquet for plasma physics 
computations to illustrate advantages and expressiveness of Lucid over an 
equivalent solution written in Fortran. 

4. GIPL, 1999 |Paq99| . 

• All Lucid dialects can be translated into this basic form of Lucid, GIPL 
through a set of translation rules. (GIPL is in the foundation of the exe- 
cution semantics of GIPSY and its GIPC and GEE because its AST is the 
only type of AST GEE understands when executing a GIPSY program). 

5. RLucid, 1999, |GP99j 

• A Lucid dialect for reactive real-time intensional programming. 

6. JLucid, Objective Lucid, 2003 - 2005 

• These dialects introduce a notion of hybrid and object-oriented program- 
ming in the GIPSY with Java and Indexical Lucid and GIPL, and are 
discussed great detail in the follow up chapters of this thesis. 
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7. Lucx |WAP05j . 2003 - 2005 



• Kaiyu Wan introduces a notion of contexts as first-class values in Lucid, 
thereby making Lucx the true intensional language. 

8. Onyx |Gro04], April 2004. 

• Peter Grogono makes an experimental derivative of Lucid - Onyx to in- 
vestigate on lazy evaluation of arrays. 

9. GLU# [PK04] . 2004 

• GLU# is an evolution of GLU where Lucid is embedded into C++. 

2.2.2 Indexical Lucid 

When Indexical Lucid came into existence, it allowed accessing context properties in 
multiple dimensions. Prior Indexical Lucid, the only implied dimension was a set of 
natural numbers. With Indexical Lucid, we can have more than one dimension, and 
we can query for a part of the context (any dimensions of it). Thus, the syntactic 
definition has been amended to include an ability to specify which dimensions exactly 
we are working on. 

2.2.2.1 Streams 

Lucid variables and expressions are said to be streams of values, through which one 
can navigate using some sort of navigational operators. In the natural language 
example given earlier the operators were beforeQ, afterQ, and at{); here we begin by 
introducing firstQ and next{) (very much like in LISP). 
If the following equations holcQ 

• FIRST X = 

• NEXT X = X + 1 (like succ in LISP) 

^Note, these are initial conditions of a definition to illustrate the ideas behind the streams and 
not an actual declaration of constructs in the language one would normally write. 
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where is a stream of O's: (0, 0, 0, 0, ...). Likewise, 1 is a stream of I's, and the '+' 
operator performs pair-wise addition of the elements in the streams according to the 
imphed current dimension index. Thus, X is defined as a stream, such that: 

• xo = 0, Xi+i = Xi + l, or 

• X = (xcxi, ...) = (0, 1, ...) 
Similarly, if: 

• FIRST X = X 

• NEXT Y = Y+ NEXT X 

Y here becomes a running sum of X: 

• yo = xo; Vi+i =yi + Xi+i 

• Y = iyo,yi,...,yi,...) = (0,1, ...,1(1 + l)/2, ...) 
2.2.2.2 Basic Operators 

This section defines properties of basic Lucid operators, which were proven by Paquet 
in |Paq99| . 

Operator fby. Operator fby stands for "followed by", fby allows simply to sup- 
press dimension index and switch to another stream. As an example the previously 
shown streams X and Y can be defined as follows using fby: 

• X = FBY X + 1 = (0,l,2,...,z,...) 

• Y = X FBY Y+ nextX = (0, l,...,i(z + l)/2, ...) 

To provide an analogy to lists, we can say that that the following operators are 
equivalent: 

• FIRST and hd 

• NEXT and tl 

• FBY and cons 
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Informal Definition of first, next, fby. 

• Definitions: 

- FIRST X = {xq, Xq, ...,Xo, ...) 

- NEXT X = {Xi, X2, ...,Xi+i, ...) 

- X FBY y = (Xo, yo, yi, Vi-l: ■■■) 

• These are tlie tliree operators of tlie original Lucid. 

• Indexical Lucid lias come into existence with the ability to access an arbitrary 
element by some index i in the stream. 

Operators wvr, asa, and upon. The other three operators that are shghtly more 
complex informally defined below: 

• X WVR Y — 

if FIRST Y ^ 

then X FBY (next X wvr next Y) 
else (next X wvr next Y) 

• X ASA Y = FIRST {X WVR Y) 

• X UPON Y — 

X FBY 

(if FIRST Y j^O then (next X upon next Y) else {X upon next Y)) 

where WVR stands for whenever, asa stands for as soon as and upon stands for advances 
upon. WVR chooses from its left-hand-side operand only values in the current dimension 
where the right-hand-side evaluates to true, asa returns the value of its left-hand-side 
as a first point in that stream as soon as the right-hand-side evaluates to true. Unlike 
ASA, UPON switches context of its left-hand-side operand uf the right-hand side is true. 
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2.2.2.3 Sequentiality Problem 

With tagged-token dataflows of tlie original Lucid operators one could only define an 
algorithm with pipelined, or sequential, data flow: 

• It is wasteful use of computing resources (e.g. to compute an element i we need 
i — I, but i — 1 may never be used/needed otherwise). 

• Sequential access to the stream of values. 

2.2.2.4 Random Access to Streams 

New intensional operators arc introduced to remedy the sequentiality problem: @ 
and 7^. The operators are used as an index ^ corresponding to the current position 
that allows querying the current context, and @ is intensional navigation to switch 
the context. With @ and #: 

• the computation is defined according to a context (here a single integer) , 

• Lucid is no longer a data-flow language and is on the road to intensional pro- 
gramming, and 

• the previously introduced intensional operators can be redefined in terms of the 
operators # and @. 
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In terms of the three original operators of first, next, and fby the operators @ and 
# are defined as follows: 



Definition 1 

# = FBY (# + 1) 

X@Y = if r = then first X else (next X)@{Y - 1) 

Both X and Y in the above definition are variable streams, and their current values are 
determined by their current context at the time of evaluation. To redefine the meaning 
of @ and # Paquet uses the denotational form, with the following proposition: 

Proposition 1 

(1) [#]. = i 

(2) [X@Y], ^ [X][y], 

where (1) means the value of ^ at the current context i is i itself (i.e. wc query the 
value of our current dimension), and (2) says that evaluate Y at the current context 
i and then use F as a new context for X. 



2.2.2.5 Definition of Lucid Operators By Means of @ and # 

First we present the definition of the operators via @ and # denoted in monospaced 
font, and then we will provide their equivalence to the original Lucid operators, 
denoted as small caps. 

Definition 2 

(1) first X = xm 

(2) next X = X@(# + 1) 

(3) X f by F = if # = then X else r@(# - 1) 

(4) X mrY = X@T where 

T = U fhy U@{T+1) 

[/ = if y then # else next U 

end 

(5) X asa y = first {X wvr Y) 

(6) X upon Y = X@W 

where = f by (if Y then {W + 1) else W) end 
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op 


::= intensional-op 
data- op 




intensional-op 


::= i-unary-op 
i-binary-op 




i-unary-op 
i-binary-op 


::= first next prev 
::= fby wvr asa upon 




data- op 


::= unary- op 
binary- op 




unary- op 
binary-op 

arith-op 


::= ! — iseod 

::= arith-op 

rel-op 

log- op 
■■■■= +1-1*1/1% 




rel-op 




I = 


log- op 


::= && 1 1 1 





Figure 1: Concrete Indexical Lucid Syntax 

E ::= id 

I E{Ei,...,En) 

I if E then E' else E" 

I i^E 

I E@E'E" 

I E where Q 

Figure 2: GIPL Expressions 

2.2.2.6 Abstract Syntax of Lucid 

Abstract and concrete syntaxes of Lucid for expressions, definitions, and operators 
are presented in Figure |2| Figure |3| and Figure [T]for both Indexical Lucid and GIPL. 

2.2.2.7 Concrete GIPL Syntax 

The GIPL is the generic programming language of all intensional languages, defined 
by the means of only two intensional operators - @ and #. It has been proven that 
other intensional programming languages of the Lucid family can be translated into 
the GIPL |Paq99| . The concrete syntax of the GIPL is presented in Figure |4j It 
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Q ::= dimension id 

I id = E 

I id{idi,id2, ...,idn) = E 

I QQ 

Figure 3: GIPL where Definitions 



id 

E(E, . . . ,E) 
E[E, . . . ,E] (E, . . . ,E) 
if E then E else E fi 
# E 

E [E:E] 
E @ E 

E where Q end; 
[E:E, . . . ,E:E] 
iseod E; 

dimension id, . . . ,id; 
id = E; 

id(id, . . . . , id) = E; 
id[id, . . . ,id] (id, .... , id) 
QQ 



#LUCX 
#GIPL 



#GIPL 
#LUCX 



= E; 



#LUCX 

#1NDEX1CAL 



#LUCX 
#GIPL 



Figure 4: Concrete GIPL Syntax 



has been amended to support the isoed operator of Indexical Lucid for completeness 
and influenced by the productions from Lucx |WAP05j to allow contexts as first-class 
values while maintaining backward compatibility to the GIPL language designed by 
Paquet in |Paq99| . 

2.2.2.8 Semantic Rules 

Paquet's PhD thesis |Paq99| presents details of the operational semantics of GIPL 
recited here for the unaware reader with a brief description. Figure |5] provides initial 
operational semantic rules for Indexical Lucid in Hoare Logic |Moe04^ IHJ02j . Later 
on, these rules are extended to support free Java methods and Java objects in JLucid 
and Objective Lucid respectively in Chapter [3j 
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Notation 



• T> represents the definition environment where all symbols are defined (a dic- 
tionary of identifiers). 

• V^V \- E : a represents current context of evaluation (a set of dimensions V) 

and the dictionary that yields a specified result a under that context given 
expression E. 

• const, op, dim, func, and var represent what kind of construct types are put 
into T> as constants, operators, dimensions, functions, and variables respectively. 

• the Exid type of rules place different identifier types listed above into the defi- 
nition environment T>. 

• the remaining Exyz-style rules correspond to the execution (or rather application 
of) of the operators, functions, and conditionals to their argument expressions 
given the definition of them in V and the current context. Thus, Eop speci- 
fies application of a defined operator function / in the current context to its 
arguments (usually one for unary operators and two for binary); Efct applies 
the named function to its arguments translating the formal arguments to ac- 
tual; Ec^ and Ecp correspond to conditional evaluation of the then and else 
branching clauses; Eat and Etag correspond to the universal intensional oper- 
ators @ and # for switching of and querying for the current context; and E^ 
corresponds to the scope definition marked by the where clause. 

• the Q-style rules allow definitions within the scope of the dimension Qdim and 
variable identifier Qid types and their composition. 
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"Di^id) = (const, c) 
V,T"rid:c 

P(»d) = (op,/) 
V,'P\- id: id 

V{id) = (dim) 
V,V\- id: id 

V{id) = (func, idi,E) 

V,'P'r id: id 

V{id) = (var, E) V,V \- E : v 
V,V\- id:v 

V,r\-E:id V{id) = {op,f) V,V\-Ei:vi 
V,V \- E{Ei, . . . , En) : f{vi,...,Vn) 

V,V\-E:id V{id) = {func, idi, E') V,V \- E'[idi <- Ei] : v 



V,V h E{Et_,. 

V,r\-E:true V,r\-E':v' 
V,V\- ii E then E' else E" : v' 

V,r\-E: false V,Vh- E" : v" 
V,V\- if E then E' else E" : v" 

V,V\-E:id I? ( id) = (dim) 
V,V^ #E :V{id) 

V,r\-E':id X>(jd) = (dim) 



■ ■,E„):v 



V,VhE":v" V,V^id^v"]\- E :v 



V,V\- E @E' E" : V 

V,V\- Q : V',V' V',V' \- E :v 
V,V\- E where Q : v 

©.P h dimension id : V 1; [id (dim)], Vt [id 0] 

V,r\-id=E : V^idi-* {var,E)],r 

V,V\-Q : V',V' v',r' \- Q' :V",r" 
V,PhQQ' : V",P" 

Figure 5: Operational Semantics of GIPL 
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2.2.2.9 Examples of Lucid Programs 

Two simple examples of Lucid programs are presented. The examples demonstrate 
absence of iterative/sequential operation as opposed to the traditional imperative 
programming languages. 

Natural Numbers Problem An example program in Indexical Lucid that yields 
44 as the result is in Figure |6} The way the program is expanded using the re- 
definitions of the Lucid operators, such as f by, employing @ and # in GIPL is shown 
in Figure [7} 



N a.d 2 
where 

dimension d; 

N = 42 fby.d (N + 1); 

end; 



Figure 6: Natural numbers problem in Indexical Lucid. 



N S.d 2 
where 

dimension d; 

N = if (#.d <= 0) then 42 else (N + 1) O.d (#.d - 1) fi; 

end; 



Figure 7: Natural numbers problem in GIPL. 



The Hamming Problem This example (see Figure [8j) illustrates the simple use 
of functions in Lucid. 



H 

where 

H = 1 fby merge (merge (2 * H, 3 * H) , 5 * H) ; 

merge (x, y) = if (xx <= yy) then xx else yy 
where 

XX = X uponCxx <= yy) ; 
yy = y uponCyy <= xx) ; 

end; 

end; 



Figure 8: Indexical Lucid program implementing the merge () function. 
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2.2.3 Lucid Now 



To summarize, Lucid is a functional programming language where a variable (stream), 
a function, a dimension, or even entire context can be a first class value (i.e. can 
viewed and manipulated as data). Lucid provides operators, such as @ and to 
navigate within dimensions and switch contexts. The language also exhibits the 
eductive execution model (demand-driven distributed computation) that augments 
the semantics with a warehouse (intensional value cache) and its consistencjr] 



2.3 Hybrid Programming 

There have been previous approaches to couple intensional or functional and imper- 
ative and object-oriented paradigms prior to this work. Some recent related work on 
the same issue is presented in |BM96[ IPKn4[ IMSOlj ISMn2j with the |PKn4j being the 
most relevant. The two major approaches of addressing the 00 issue are - either 
(1) to extend Lucid to become object-oriented or objects-aware or (2) make a host 
imperative language be extended to embed Lucid. The authors of |PK04j chose the 
latter by extending GLU-with-C to GLU#-with-C++, whereas this work approaches 
the problem from Lucid to Java. This means a Lucid program is the main one driv- 
ing the computation. We will briefly consider the following approaches to the hybrid 
programming: 

• ML< 

• FC++ 

• GLU 

• GLU# 



^Paquet defines the augmented operational semantics in |Paq99| and Tao implements its first 
incarnation in GIPSY |Tao04j . This work has an impact on this aspect by introducing the side 
effects with the imperative languages, which will be discussed later. 
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2.3.1 ML< 

ML< |BM96j is a system introduced in 1996 that proposed to marry OOP and func- 
tional paradigms using their own language and providing the details of the predicative 
and decidable typing rules and operational semantics of such a system. Their main 
goal is to be able to induce implicit polymorphism of functional languages in objects. 
They do not extend an existing functional language with the 00 capabilities, in- 
stead they reinterpret all data types as either abstract or concrete classes and use the 
dynamic dispatch, a typical 00 feature, on run-time types. 

2.3.2 FC++ 

FC++ [ MSOll ISM02j tries to promote the functional paradigm in C++. FC++ is a 
library add-on to enable higher-order polymorphic functions in a novel use of C++ 
type inference that is not very complex and is still expressive. FC++ adds support for 
both parametric and subtype polymorphism policies for functions in order to be able 
to fit FC++ functions within the C++ object model and pass higher-order functions 
as parameters. The FC++ functions are kept as objects called functoids and use 
a reference counter machinery for allocation and de-allocation. Closures in FC++ 
(operation on a some state and the state itself) can automatically be created during 
functoid object creation, but their "closing" of that state is not automatic and the 
state values have to be passed explicitly during the creation process. The library also 
adds a set of functional operators from the Haskell Standard Prelude. FC++ comes 
more from the OOP-to-functional point of view and conforms with standard software 
engineering design patterns and is suitable for the common 00 tasks. 

2.3.3 GLU 

GLU was the most general intensional programming tool recently available |JD96] . 
However, experience has shown that, while being very efficient, the GLU system 
suffers from a lack of flexibility and adaptability |Paq99| . Given that Lucid is evolving 
continually, there is an important need for the successor to GLU to be able to stand 
the heat of evolution |Paq99| . The two major successors of GLU are the GIPSY and 
GLU# systems. 
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Eduction 

The earlier mentioned notion of eduction was first introduced by the GLU compiler. 
GLU supports so-called tagged-token demand-driven dataflow where data elements 
(tokens) are computed on demand following a datafiow network defined in Lucid. 
Data elements fiow in the normal fiow direction (from producer to consumer) and 
demands fiow in the reverse order, both being tagged with their current context of 
evaluation. This form of lazy computation is inherited by GIPSY from GLU. 

2.3.4 GLU# 

GLU# |PK04] is a successor of GLU, which enables Lucid within C++. The authors 
argue for the embedding small functional/intensional-language pieces of Lucid into 
C++ programs allowing lazy (demand-driven) evaluation of arrays and functions 
thereby making Lucid easily accessible within a popular imperative programming 
language, such as C++. Because GLU# appeared quite recently (2004) to when this 
work was written, its success compared to GLU is yet to be evaluated; however, it 
seems to suffer from the same inflexibility GLU did and targets only C++ as a host 
language. 

2.4 Compiler Frameworks 

A signiflcant number of compiler frameworks emerged for the past decade. All try to 
enable compilation of more than one language, either hybrid or not, in an uniform 
manner. Some frameworks or libraries became "frozen" (i.e. non-extendable) and 
flxed to a speciflc set of languages, some other ones were build with the extension 
in mind, so it is relatively easy to "plug-in" yet another compiler into the system 
(a collection of compilers and the necessary tools) with minimum integration work 
required. A brief overview of different compiler frameworks is given next: 

• GLU tried to accommodate Fortran, C, and Lucid in one system, but was made 
so inflexible |Paq99| that it would take a signiflcant effort to extend it and add 
other languages to the system. 

• GLU# merges Lucid and C++; however, makes no provisions for extension to 
other languages on either intensional or imperative side. 
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• Microsoft .NET can also be thought of a commercial heterogeneous compiler 
framework (it is more than a compiler framework, but our focus is on compil- 
ers) that allows easy cooperation and application development between different 
language models, such as C#, C++, Visual Basic, and Assembly in a homo- 
geneous environment. However, none of these languages have natively any of 
the intensional or functional capabilities, so no native debugging support or 
other tools exist, even if one starts using FC++ or GLU^^ in this environment. 
Despite the fact that all programs can be compiled into the common bytecode, 
the debugging tools have to be aware of the functional paradigms on a higher 
level and they are not (at least at this writing). 

• The GNU Compiler Collection (GCC) can also be said as a compiler framework 
from the free software |CP05j . It supports C, C++, Objective-C, Objective- 
C++, Java, Fortran, and Ada. Again, these languages are more of an imperative 
nature, but it is far easier to add new language into GCC than to Microsoft 
.NET due to its openness. 

• Finally, the GIPSY presents the GIPC framework that is designed for expan- 
sion and integration of the intensional and imperative (and later functional) 
languages. This is presented through the rest of this thesis. 
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Figure 9: The GIPSY Logo representing the distributed nature of GIPSY. 

2.5 General Intensional Programming System 
2.5.1 Introduction 

GIPSY is broadly presented in |WP(;n3t [LuOit IPWOSj . and others. Please refer to 
the online resources |RG05al IPWOSt IRGOSb] to obtain the most current status of the 
project. GIPSY is primarily implemented in Java. General GIPSY architecture is 



presented in Figure 10 The essence behind GIPSY is demand-driven computation 
support for the intensional programming languages, e.g., Indexical Lucid, Tensor 
Lucid |Paq99| , etc. 

The GIPSY consists in three modular sub-systems: the General Intensional Pro- 
gramming Language Compiler (GIPC); the General Eduction Engine (GEE), and 
the Intensional Run-time Interactive Programming Environment (RIPE). The sub- 
systems have to be modular so that one implementation of parts of them or the whole 
can be replaced by another without having major if any impact on the other modules. 
Although the theoretical basis of the language has been settled, the implementation 
of an efficient, general and adaptable programming system for this language raises 
many questions. The following sections outline the theoretical basis and architecture 
of the different components of the system. All these components are designed in a 
modular manner to permit the eventual replacement of each of its components - at 
compile-time or even at run-time - to improve the overall efficiency and productivity 
of the system |Paq99| . 

A GIPSY instance sends out little bits of work to others to compute and then 
gathers the results in distributed fashion. Of course, synchronization, latency toler- 
ance, and maximum utilization of resources are primary goals for the system to be 
productive. Unlike in most programming language models (see |ST98] ) considered 
for parallel computation, in GIPSY several key concepts are considered: 
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Figure 10: Structure of the GIPSY 

• Thread-Level Parallehsm (TLP) 

• Stream-Level Parallelism (SLP) 

• Cluster-Level Parallelism (CLP) 

gipsy's parallelism granularity takes into account the amount of TLP, SLP, and 
CLP available. TLP determines the maximum number of threads that should or can 
be created when a Lucid program is being executed. In other words, TLP defines on 
how many pieces of terminal computational work we can chop a big job into. The 
goal, as far as programming is concerned, is to program for infinite TLP, and later 
adjust (load-balance) at run-time to the actual amount of SLP. SLP determines the 
maximum number of streams available to execute the threads. Here, by "streams" we 
mean processors but, with the invention of multithreaded CPUs for a single processor, 
there may be several thread streams available in parallel, and hence a more general 
notion of SLP. The amount of SLP is machine-dependent and has to be discovered at 
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run-time on remote machines. If a job is to be run on a single machine, GIPSY tries 
to maximize SLP utilization, providing just enough TLP for the machine in question 
with the design goal of always assuming infinite TLP. Then load-balancing comes 
into play. CLP takes GIPSY to another level — distributed computing, involving 
utilization of SLP of the machines across the network nearby or across the globe over 
the Internet. 

NOTE: the Lucid family of languages has also a notion of streams that refers 
to Lucid variables that evaluate in multiple contexts. Every Lucid stream (e.g. a 
variable) can potentially be evaluated on any hardware stream available, but it is 
important not to confuse the two kinds of streams. The reason for the existence of 
the two notions is that both terms were used independently in each field. Now that 
parallel architectures and language models such as Lucid came into proximity, the 
terms clash. 



2.5.2 Goals 

The system has to withstand the evolution of the tools, languages, and underlying 
platforms, thus be flexible and adaptable to the changes. That is one of the most 
important and stringent requirements put on the development of GIPSY |Paq99| . 
Other subordinate requirements in compiler design, run-time system, communication, 
and user interfaces are presented in detail throughout the follow up sections. 



2.5.3 General Intensional Programming Compiler 



GIPSY programs are compiled in a two-stage process (see Figure |33| page 65). First, 
the intensional part of the GIPSY program is translated in Java, then the resulting 
Java program is compiled in the standard way. 

The source code consists of two parts: the Lucid part that defines the intensional 
data dependencies between variables and the sequential part that defines the granular 
sequential computation units (usually written in any imperative language, e.g. C or 
Java). The Lucid part is compiled into an intensional data dependency structure 
(IDS) describing the dependencies between each variable involved in the Lucid part. 
This structure is interpreted at run-time by the GEE following the demand propa- 
gation mechanism. Data communication procedures used in a distributed evaluation 
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Figure 11: Initial Conceptual Design of the GIPC 
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of the program are also generated by the GIPC according to the data structures 
definitions written in the Lucid part, yielding a set of communication procedures 
(CP). These are generated following a given communication layer definition such as 
provided by RFC (or rather RMI since GIPSY is implemented in Java), CORBA, 
Jini, or the WOS |BKU98j . The sequential functions defined in the second part of 
the GIPSY program are translated into imperative code using the second stage im- 
perative compiler syntax, yielding imperative sequential threads (ST). Intensional 
function definitions, including higher order functions, will be fiattened using a well- 
known efficient technique |Ron94t Paq99| . The closures in the higher order functions 
case are still applicable because the function state and the operation on it are cor- 
rectly passed to the functions by expanding and using function definitions inline. The 
insignificant limitation here is that self-referential closures for such functions cannot 
be made. The function elimination in GIPSY pertinent to some of these aspects was 
implemented by Wu in |Wu02j . 



The Figure 11 presents the initial conceptual design of the GIPC. Based on this 
design, the GIPSY module integration and the development of the STs and CPs 
support has begun. Later on the design was refined in |PGW04| IMPOSaj and its 
latest reincarnation is shown in Figure 38 in Chapter |4| page 77 thus, the evolution 
description is delayed until then. 

Prior this work, GIPC supported only two Lucid dialects: GIPL and Indexical 
Lucid. The initial GIPC compiler was implemented by Chun Lei Ren in |Ren02] . 
and the translation of the Indexical Lucid into GIPL and the semantic analysis was 
implemented by Aihua Wu in |Wu02] . A large integration and re-engineering effort 
went into GIPC to approach it to the goals of the GIPSY (see Section 2.5.2) and add 
more compilers for investigation of the underlying language models. The results of 
this effort are presented in the Design and Implementation chapter (Chapter |4]). 



2.5.4 General Eduction Engine 

The GIPSY uses a demand-driven model of computation, which is based on the prin- 
ciple is that certain computation takes effect only if there is an explicit demand for it. 
The GIPSY uses eduction, which is demand-driven computation in conjunction with 
an intelligent value cache called a warehouse. Every demand can potentially generate 
a procedure call, which is either computed locally or remotely, thus eventually in 
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Figure 12: Conceptual Design of the GEE 



parallel with other procedure calls. Every computed value is placed in the warehouse, 
and every demand for an already-computed value is extracted from the warehouse 
rather than computed again and again (demands that may have side effects, e.g. if 
we cache results of STs, shall not be cached). Eduction, thus, reduces the overhead 
induced by the procedure calls needed for the computation of demands sequentially. 



Figure 12 describes the internal conceptual structure and functioning of the GEE. 

The GEE itself is composed of three main modules: the executor, the intensional 
demand propagator (IDP), and the intensional value warehouse (IVW). First, the 
intensional data dependency structure (IDS, which represents GEER) is fed to the 
demand generator (DG) by the compiler (GIPC). This data structure represents the 
data dependencies between all the variables in the Lucid part of the GIPSY program. 
This tells us in what order all demands are to be generated to compute values from 
this program. The demand generator receives the initial demand, that in turn raises 
the need for other demands to be generated and computed as the execution progresses. 
For all non-functional demands (i.e. demands not associated with the execution of 
sequential threads (ST)), the DG makes a request to the warehouse to see if this 
demand has already been computed. If so, the previously computed value is extracted 
from the warehouse. If not, the demand is propagated further, until the original 
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demand resolves to a value and is put in the warehouse for further use. This type 
of warehousing was introduced by GLU due to its distributed nature to cut down on 
communication costs, but it can certainly be applicable to any functional language, 
such as LISP, Scheme, Haskell, ML and others to improve efficiency even on a single 
machine provided there are no any side effects whatsoever. The garbage collector can 
run on the background to clean up old function-parameters- values tuples periodically, 
and given that the large amounts of memory are cheap these days functional languages 
may gain much more popularity with the increased performance. 

For functional demands (i.e. demands associated with the execution of a sequen- 
tial thread), the demands are sent to the demand dispatcher (DD) that takes care 
of sending the demand to one of the workers or to resolve it locally (which normally 
means that a worker instance is running on the processor running the generator pro- 
cess). If the demands are sent to a remote worker, the communication procedures 
(CP) generated by the compiler are used to communicate the demand to the worker. 
The demand dispatcher (DD) receives some information about the liveness and effi- 
ciency of all workers from the demand monitor (DM), to help it make better decisions 
in dispatching the demands. 

The demand monitor, after some functional demands are sent to workers, starts 
to gather various types of information about each worker, including, but not limited 
to: 

• liveness status (is it still alive, not responding, or dead) 

• network link performance 

• response time statistics for all demands sent to it 

These data points are accessed by the DD to make better decisions about the load 
balancing of the workers, and thus achieving better overall run-time efficiency. 

Bo Lu was the first one to do the original design of the GEE framework |Lu04j 
and investigate its performance under threaded and RMI environments. She also 
introduced the notion of the Identifier Context (IC) classes - demands converted 
into Java code and using Java Reflection |Gre05] to compile, load, and execute them 
them at run-time. She also contributed the flrst version of the interpreter-based 
execution engine. Next, Lei Tao contributed the flrst incarnation of the intensional 
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value warehouse and garbage collection mechanisms in |Tao04j based on the popular 
scientific library called NetCDF. The author of this thesis put an effort to modularize 
these all and make them easier to extend and customize. He also provided the initial 
GEE application to start available network services. The GEE was also made aware 



of the STs and CPs as well as the new type system, described in Section 4.1.1.5 



Further, Emil Vassev |VP05j produced a very general and functional framework for 
demand migration and its implementation. Demand Migration System (DMS) that 
supports among other things Jini, CORBA, and .NET Remoting for fault-tolerant 
demand transportation system, a part of the Demand Dispatcher. The DMS is still 
pending integration as of this writing. 



2.5.4.1 Demand Propagation Resources for the GEE 

The IDF generates and propagates demands according to the data dependence struc- 
ture (DFR, now renamed to GEER in |WFG03] ) generated by the GIFC. If a demand 
requires some computation, the result can be calculated either locally or on a remote 
computing unit. In the latter case, the communication procedures (CF) generated by 
the GIFC are used by the GEE to send the demand to the worker. When a demand is 
made, it is placed in a demand queue, to be removed only when the demand has been 
successfully computed. This way of working provides a highly fault-tolerant system. 
One of the weaknesses of GLU is its inability to optimize the overhead induced by 
demand-propagation. The IDF will remedy to this weakness by implementing various 
optimization techniques: 

• Data blocking techniques used to aggregate similar demands at run time, which 
will also be used at compile-time in the GIFC for automatic granularization of 
data and functions for data-parallel applications 

• The performance-critical parts (IDF and IVW) are designed as replaceable mod- 
ules to enable run-time replacements by more efficient versions adapted to spe- 
cific computation-intensive applications 

• Certain demand paths identified (at compile-time or run-time) as critical will 
be compiled to reduce their demand propagation overhead 

• Extensive compile-time and run-time rank analysis (analysis of the dimension- 
ality of variables) |Dod96j . 



30 



2.5.4.2 Synchronization 
Distributed vs. Parallel 



It is important to make a distinction between parallel and distributed computing. In 
parallel computing, SLP matters and latency tolerance for memory references with 
mostly UMA (uniform memory access) characteristics, whereas in distributed com- 
puting communication is much more expensive (and perhaps even prohibitive) and 
CLP matters as well. This setup largely exhibits NUMA (non-UMA) characteristics 
(see |Pro03b] ) and latency tolerance (and so also fault tolerance) has a higher sig- 
nificance. This greatly impacts the way we synchronize in parallel and distributed 
worlds. 



Synchronization in Distributed Environment A distributed environment is a 
very popular domain these days, so we'll start with it first. Typically, the network 
is the scarce resource and is the bottleneck for a distributed application because it 
implies communication (e.g., MPI), which is often unacceptable. Therefore, many 
distributed applications choose not to communicate at all or communicate very little 
through message passing. This implies blocking on waiting for the network requests 
to propagate, i.e. network latency. 

Synchronization in Parallel Environment Synchronization in a parallel envi- 
ronment is more fine-grained, often at the hardware level (e.g., a full/empty bit in 
memory cells). Java does not give us control over such synchronization, so we have to 
rely on the JVM built for an architecture that has such synchronization. The JVM 
has to be developed to make use of the full/empty bits that are usually represented 
as future variables |Pro03bl IJA03j in the languages specifically designed for parallel 
computing. 



Secure Synchronization 

Secure synchronization is especially pertinent in a distributed environment. Like any 



act of communication within worker-generator architecture (see Section 3.3.3.4) and 
a warehouse (Figure 33 Section 2.5.4.1), synchronization has to be secure to avoid 
(a) over- demanding, (b) incorrect results sent back, (c) loss of results and demands, 
and (d) poisoning the warehouse with wrong data. Secure synchronization implies 
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fault tolerance. In GIPSY, we will rely on Java's RMI and Jini over JSSE for secure 
communication in a distributed environment, using Java's synchronization primitives 



(see Section 2.5.4.2) to achieve the goal of secure synchronization. Thus, the relia- 
bility and accountability of the results of a GIPSY program are dependent on these 
properties of underlying Java Runtime Environment (JRE) and the communication 
protocols used. 



Implicit vs. Explicit Synchronization 

One of the productivity metrics of a software completing its task on time, is the effi- 
ciency of development of (see |Pro03c] ) such a software, i.e., the amount of program- 
mer's effort required to create and debug the software. This is essentially a metric, 
called time-to-solution (TTS) |Pro03cj : from creation until the end result (e.g. com- 
pletion of some scientific computation). The goal is to minimize TTS. One way to 
achieve this is ease of programming. As the proportion of the work done by the com- 
piler increases, so does the reliability of the code, but we target scientific researchers, 
not just programmers. Scientific researchers from math and physics should not care 
about these issues and, thus, just be concerned mastering the basics of Lucid. There- 
fore, the programmer has to be freed from taking care of synchronization explicitly, 
which a source of bugs and inefficiency of programming (e.g., using Java's synchro- 
nization primitives, such as synchronized. Object .waitO, Object .notifyO, and 
Object. notifyAllO, |Fla97] ). The programmer should rather focus on the problem 
being solved and let the compiler/run-time system deal with the synchronization pain. 
The GIPSY system, built around the Lucid family, advocates implicit synchronization 
either by wrapping around the Java's synchronization primitives or through the com- 
munication synchronization and data dependencies (although a complete discussion 
is beyond the scope of this thesis, see |Lu04l IVPOSj ). 



2.5.5 Run-time Interactive Programming Environment 

The RIPE is a visual programming aid to the run-time environment (GEE) enabling 
the visualization of a datafiow diagram corresponding to the Lucid part of the GIPSY 
program, source code editing, launching the compilation and execution of GIPSY 



programs. The original conceptual design of RIPE |Paq99| is illustrated in Figure 13 



The user's points of interaction with the RIPE at run-time vary in the following ways: 
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Figure 13: Conceptual Design of the RIPE 

• Enable interactive editing of GIPSY programs via a variety of editors (textual, 
graphical, web). 

• Dynamic inspection of the IVW. 

• Modification of the input/output channels of the program. 

• Recompilation of the GIPSY programs. 

• Modification of the communication protocols. 

• Swapping of the parts of the GIPSY itself (e.g. garbage collection, optimization, 
warehouse caching etc. strategies). 

Because of the interactive nature of the RIPE, the GIPC is modularly designed 
to allow the individual on-the-fly compilation of either the IDS (by changing the Lu- 
cid code), CP (by changing the communication protocol), or ST (by changing the 
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sequential code). Such a modular design even allows sequential threads to be pro- 
grams written in different languages (for now, we are concentrating on Java sequential 
threads, but a provision is made for easy inclusion of other languages with the GICF, 



Section |4.1.1.lD . 

The RIPE even enables the graphic development of Lucid programs, translating 
the graphic version of the program into a textual version that can then be compiled 
into an operational version through a DFG generator of Yimin Ding |Din04j . However, 
the development of this facility for graphical programming posed many problems 
whose solution is not yet settled, for example representation of the STs and CPs in 
the DFG nodes. An extensive and general requirements analysis will be undertaken, 
as this interface will have to be suited to many different types of applications. There 
is also the possibility to have a kernel run-time interface on top of which we can plug- 
in different types of interfaces adapted to different applications, such as stand-alone, 
web-, or server-based. 
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2.6 Tools 



This section presents a brief description of a variety of tools that helped most with 
the implementation aspects of this work. 

2.6.1 Java as a Programming Language 

The primary implementation language of GIPSY is Java. This includes using Java's 
Reflection, JNI, and JUnit frameworks and packages. We have chosen to implement 
our project using the Java programming language mainly because of the binary porta- 
bility of the Java applications as well as its facilities, for e.g. memory management 
and communication tasks, so we can concentrate more on the algorithms instead. 
Java also provides built-in types and data-structures to manage collections (build, 
sort, store/retrieve) efficiently |Fla97l iMicOSbj . There is also source code written in 
other languages in the main GIPSY repository. This includes LEFTY code for DFG 
generation and the code of the test intensional programs in various Lucid dialects. 
The Java versions supported by GIPSY are 1.4 and 1.5. The GIPSY will no longer 
build on 1.3 and earlier JDKs. 

2.6.1.1 Java Reflection 

Java Reflection Framework j ava . reflect . * |Gre05j allows us to load/query /discover 
a given class for all of its API through enumeration of constructors, flelds, methods, 
etc. at run-time. This is incredibly useful for dynamic loading and execution of 
our compilers, identifler context classes, and sequential threads on local and remote 
machines. 

The basic API from the reflection framework used in the implementation of 
GIPSY is the Class class that allows getting arrays of declared Method objects 
through the getDeclaredMethodsO call that will become the STs at the end, then 
for each Method the reflection API allows getting parameter and return types via 
getParameterTypes and getReturnTypeO calls, which will become the CPs. The 
Class .newInstcinceO method allows instantiating an object off the newly gener- 
ated class. Likewise, an enumeration of Constructor objects is acquired through the 
Class. getConstructorsO call. Constructors in Java are treated differently from 
methods because they are not inherited and don't have a return type (except that 
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the type of the object they create). We still need to enumerate them to allow Objec- 
tive Lucid programs to use the constructors, default or non-default, directly, so we 
can get a handle on them similarly to STs. 

2.6.1.2 Java Native Interface (JNI) 

The Java Native Interface (JNI) |Ste05j is very useful for the thread generation com- 
ponent of the GIPC. We rely on JNI to increase the number of popular imperative 
languages in which the sequential threads could be written. Developers use the JNI 
to handle some specific situations when an application cannot be written entirely in 
Java, e.g. when the standard Java classes do not provide some platform-dependent 
features an application may require, or use a library written in another language be 
accessible to Java applications, or for performance reasons a small portion of a time- 
critical code has to be written say in C or assembly, but still be accessible from a 
Java application |Ste05] . In GIPSY, the second and third of the listed cases are most 
applicable (e.g. to adopt GLU programs). The JNI will allow us to avoid Lucid-to-C 
or Lucid-to-C++ type matching as we can do it all through Java and maintain only 
Lucid-to-Java type mapping table. 

The JNI is made so that the native and Java sides of an application can pass back 
and forth objects, strings, arrays and update their state on either end |Ste05j . The 
JNI is bi-directional, i.e., allows Java to use the native libraries and applications and 
provide access to Java libraries from the native applications. 

The general methodology of creating a JNI application say that interacts with a 
C implementation is done in six steps |Ste05j : 

1. Write a Java code with a native method to be implemented in C, the mainO, 
and the dynamic loading statement for a library (to be compiled in the next 
steps). 

2. Compile the Java code with javac and produce a .class file. 

3. Create a C header .h file from the compiled . class file by calling javah. This 
header file will provide the necessary #include directives along with the C-style 
prototype declaration of the native method. 

4. Next, write the implementation of the function in regular C in a .c file. 
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5. Then, create a shared hbrary by compihng the . h and . c files with a C compiler. 

6. Run the application regularly with the JVM (Java). 

2.6.1.3 JUnit 

JUnit is an open-source Java testing framework used to write and run automated 
repeatable unit tests in a hassle-free manner |GB04] . The goal is to sustain application 
correctness over time, especially when undergoing a lot of integration efforts. JUnit 
is designed with software architecture patterns in mind and follows best software 
engineering practices. It encourages developers to write tests for their applications 
that withstand time and bit rot. 

The main abstract class is TestCase that follows the Command design pattern 
that implements the Test interface. This class maintains the name of the tests (if 
it fails) and defines the run() method that has to be overridden to do the actual 
testing work. The default Template Method run () simply does three things: setUpO, 
runTestO, and tearDownO. Their default implementation is to do nothing, so a 
developer can override them as necessary. Then, to collect the test results they apply 
Collecting Parameter pattern. They use the TestResult class for that. 

JUnit makes a distinction between errors and failures in the following way: errors 
to JUnit are mostly unexpected run-time or regular exceptions, whereas failures are 
anticipated and are tested for using assertion checks. The errors and failures are 
collected for further test failure reporting. 

To run tests in a general manner from the point of view of the tester, the test 
classes have with a generic interface using the Adapter pattern. JUnit also offers 
a pluggable selector capability via the Java Reflection API |Gre05j . The TestSuite 
class represents a collection of tests to run. In the GIPSY, the Regression application 



(see Section 5.1) comprises concrete implementation of such a test suite that tests 
most of the feasible functionality of the GIPC and GEE modules. See more details 
of application of JUnit to the GIPSY in Chapter |5| 

2.6.2 javacc — Java Compiler Compiler 

JavaCC jVCOSj . accompanied by JJTree, is the tool the GIPSY project is relying on 
since the first implementation |Ren02j to create Java-language parsers and ASTs off 
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a source grammar files. The Java Compiler Compiler tool implements the same idea 
for Java, as do lex/yacc |Lou97j (or flex/bison) for C - reading a source grammar 
they produce a parser that complies with this grammar and gives you a handle on 
the root of the abstract syntax tree. The GIPL, Indexical Lucid, JLucid, Objec- 
tive Lucid, PreprocessorParser, and DFGGenerator parsers are generated with the 
JavaCC/JJTree parser generation tools. JavaCC is a LL(K) |Lou97j parser generator, 
so the original GIPL and Indexical Lucid grammars and the new grammars had to 
be modified to eliminate or avoid the left recursion. 

2.6.3 MARF 

Modular Audio Recognition Framework (MARF) library |MCSN05] provides a few 
useful utility and storage classes GIPSY is using to manipulate threads, arrays, option 
processing, and byte operations. Despite MARF's belonging to a voice/speech/natural 
language recognition and processing library, it contains a variety of useful utility mod- 
ules for threading and options processing. 

2.6.4 CVS 

For managing the source code repository the Concurrent Versions System (CVS) 
[BddzzP+05j is used. The CVS allows multiple developers work on the up-to-date 
source tree in parallel that keeps tracks of the revision history and works in an trans- 
actional manner. The author produced a mini- tutorial on the CVS |Mok03aj for the 
GIPSY Research and Development team, which contains the necessary summary for 
the team to work with the project repository. 

While CVS has a comprehensive set of commands, the basic set includes: 

• init to initialize the repository 

• checkout or co to checkout the source code tree from the repository to a local 
directory 

• update or up to make the local tree up-to-date with the one on the server 

• add to schedule a new file inside the existing local checkout for addition to the 
repository 
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• remove to schedule a new file inside the existing local checkout for removal from 
the repository 

• commit to upload the changes done locally to the server 

• dif f to show the differences between the local and the server versions of the 
tree 

2.6.5 Tomcat 

Apache Jakarta Tomcat |Fou05] is an open-source Java application servlet and server 
pages container project from Apache Foundation to run web Java-based applications 
written in accordance with the Java Servlet and JavaServer Pages |Mic05at IMicOSc] 
specifications developed by Sun Microsystems. Tomcat powers up the web front end 
to GIPSY to test intensional programs online. The web frontend is represented by 
the WebEditor servlet as of this writing a part of RIPE which is discussed later in 
Chapter |4j Tomcat has an easy interface to deploy Java-based applications and their 



libraries, e.g. through a manager presented in Figure 14 

Tomcat itself consists from a variety of modules that includes implementation of 
the JSP (Jasper engine) and Servlet APIs, a webserver called Coyote, the application 
server called Catalina, and many other things for logging, security, administration, 
etc. 



2.6.6 Build System 

The gipsy's sources can be built using a variety of ways, using different compil- 
ers and IDEs on different platforms. This includes Linux Makefiles, IBM's Eclipse, 
Borland's JBuilder, Apache's Ant, and Sun's NetBeans. 

2.6.6.1 Makefiles 

Unix/Linux Makefiles are targeting all Unix systems that support GNU make (a.k.a 
gmake) |SMSPnni[Mokn5aj . Often, to compile all of the GIPSY is just enough to type 
in make and the system will be built. All Unix versions support make, and our system 
has been tested to build on Red Hat Linux 9, Fedora Core 2, Mac OS X, and 
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Solaris 9. There is a test script make-test. sh that tests whether we are deahng 
with the GNU make on UNIX systems, as this is the only make supported. 

2.6.6.2 Eclipse 

There are project files .project and . classpath that belong to this IDE from IBM 
[H+04] . The GIPSY build with this IDE properly and has its library CLASSPATH 
set. Eclipse is another open source tool available free of charge and provides extended 
tools for Java projects development, refactoring, and deployment. 

2.6.6.3 JBuilder 

There is a project file GIPSY, jpx that belongs to this IDE from Borland |Bor03j . The 
GIPSY build with this IDE properly and has its library CLASSPATH set. 

2.6.6.4 Ant 

There is a project file build. xml that belongs to this build tool from the Apache 
Foundation |Con05j . The GIPSY build with this tool properly and has its library 
CLASSPATH set. In this case build. xml is a portable way to write a Makefile in 
XML. 

2.6.6.5 NetBeans 

There is a project file nbproject .xml that belongs to this IDE from Sun |Mic04] . 
The GIPSY build with this IDE properly and has its library CLASSPATH set. 

2.6.7 readmedir 

This script generates a human-readable description of a directory structure starting 
from some directory with file listing and possibly descriptions (for this there should 
be specially formatted file README. dir in every directory traversed. The contents 
of this file will be a part of the output and is a responsibility of the directory cre- 
ator/maintainer. The output formats of the script are LTgX, HTML, and plain text. 
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2.7 Summary 



In this chapter the reader was introduced to the necessary background on the GIPSY 
project and how it is being managed starting from the Lucid language origins to 
its implementations in the GIPSY and the summary of the tools used to aid the 
advancement of the project. In the GIPSY section the three main modules were 
introduced, such as GIPC, GEE, and RIPE. While most of the remaining work has 
gone into the GIPC in this thesis, the author had to perform the necessary integration 
and adjustments to the GEE and RIPE. 
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Chapter 3 



Methodology 



This chapter focuses on the methods and techniques proposed to the solve the stated 



problems (see Section 1.1 ). The approaches described are based on three publications, 



namely |MPG05l IMP05b[ IMPOSaj . Section 3.1 introduces the JLucid language and 
all related considerations including the syntax and semantics. Next, Objective Lucid 
is introduced along with its syntax and semantics. Further, the GICF is introduced 
by providing the necessary requirements for it to exist and the way to satisfy them. 
Lastly, the summary is presented outlining the benefits and limitations of the proposed 
solutions. 



3.1 JLucid: Lucid with Embedded Java Methods 
3.1.1 Rationale 

The name JLucid comes from the GIPC component known as Java Compiler within 
the Sequential Thread (ST) Generator of the GIPSY. It subsumes all of Indexical 
Lucid and General Intensional Programming Language (GIPL) |Paq99| and syntacti- 
cally allows embedded Java code. In fact, a JLucid program looks like a partial fusion 
of the intensional and Java code segments. JLucid gives a great deal of flexibility to 
Lucid programs by allowing to use existing implementations of certain functions in 
Java, providing I/O facilities and math routines (that Lucid entirely lacks), and other 
Java features accessible to Lucid, arrays, and permits to increase the granularity of 
computations at the operator level by allowing the user to define Java operators, i.e., 
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functions manipulating objects, thus allowing streams of objects^ in Lucid. JLucid 
more or less achieves the same goals and mechanisms as provided by GLU. What we 
are proposing is a flexible compiler and run-time system that permits the evolution 
of languages through a framework approach |MP05at IPWOSj . 

3.1.1.1 Modeling Non-Determinism 

Lucid, by its nature, is deterministic, so introduction of imperative languages, such 
as Java, may allow us to model non-determinism in Lucid programs for example by 
providing access to random number generators available to the imperative languages. 
Non-determinism can also be introduced as a result of side effects from for example 
reading a different file each time an ST is invoked, or making a database query against 
a table where data regularly changes, or say by reading the current time of day value. 
Of course, a special care should be taken not to cache the results of such STs in the 
warehouse. 



3.1.1.2 Loading Existing Java Code with embedO 

In a nutshell, we want to make the following possible for the Indexical Lucid program 
in Figure [Ts] (replicated here from Chapter [2] for convenience) to become something 
as in Figure [16] or, alternatively as in Figure [17} The latter form would allow us to 
include objects from any types of URLs, local, HTTP, FTP, etc. The idea behind 
embed is to include or to import the code written already by someone and not 
to rewrite it in Lucid (which may not be a trivial task). It is not meant to adjust 
to URL's existence at run-time as all embed- referenced resources are resolved at 
compile time. We "include" the pointed-to resource and attempt to compile it where 
the original program-initiator resides. If the URL is invalid at compile time, then 
there will be a compile error and no computation will be started, embed () by itself 
does not necessarily provoke a remote function call. 

Existing Java code, in either .class or .Java form, can be loaded with embed (). 



Intuitively, we would prefer the approach presented in Figure 18 That added flexi- 



bility requires syntactical extension of Lucid and is not portable. For the program in 



more precise meaning of Java objects within Lucid is explored further in the Objective Lucid 
language, including the meaning of an object stream and how object members are manipulated (see 



for example Section 3.2 and Section 4.1.3.61. Additionally, since the actual Java objects are flattened 



into primitive types, it would be possible to access object members in parallel manner. 
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H 

where 

H = 1 fby merge (merge (2 * H, 3 * H) , 5 * H) ; 

merge (x, y) = if(xx <= yy) then xx else yy 
where 

XX = X uponCxx <= yy) ; 
yy = y uponCyy <= xx) ; 

end; 

end; 



Figure 15: Indexical Lucid program implementing the merge () function. 



#JAVA 

void merge (int x, int y) 
{ 

// Java code here 

} 

#JLUCID 
H 

where 

H = 1 fby merge(merge(2 * H, 3 * H) , 5 * H) ; 

end; 



Figure 16: Indexical Lucid program implementing the merge () function as inline Java 
method. 



H 

where 

H = 1 fby merge(merge(2 * H, 3 * H) , 5 * H) ; 
merge (x, y) = 

embedC'file: //path/to/class/Merge . class" , "merge", x, y) ; 

end; 



Figure 17: Indexical Lucid program implementing the merge () function as embedO. 



F 

where 

dimension d; 
F = foo(#d); 
where 

food) = embed("file://my/classes/Foo. class", "foo", i) ; 

end; 

end; 



Figure 18: Illustration of the embed () syntax. 
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Figure 18 to work, f oo() has to return a Java type of int, byte, long, char. String, 



or boolean, as per Table [T| page 64 A wrapper class will be created to extend from 



the Foo and implement the I Sequent ialThread interface (see Appendix B.l). Gen- 
eral embed syntax would be defined as follows: 



id(id, id, ...) : : = embed(URl , METHOD, id, id, ...); 



where id is the Lucid function name being defined that is mapped to a Java's method 
named METHOD (which may or may not be of the same name as the first id). The URI 
is pointing to either .class or .Java file. Example URI's would be: 



foo(a,b) = embedC'f ile : //f iles/Foo . Java" , "bar" , a,b) ; 
bar(a,b) = embed("http://www. java.com/Foo.class" , "foo" ,a,b) ; 
baz(a,b) = embed("f tp : //f tp . f ile . com/pub/Foo .Java" , "zee" , a,b) ; 



These declarations associate Lucid functions with Java implementations. Name 
clashes may be avoided, if necessary, by using different function names. Above, for 
example. Lucid baz() is implemented by Java zee(). 

public class <f ileiiajne>_<machine_ncLme>_<tiinestanip> 
extends my . classes . Foo 
implements ISequentialThread 
{ 

// The definition is provided later in the text 

} 



Figure 19: Generated corresponding ST to that of Figure 18 

There are several ways of making this work. We could extract either a textual or 
a bytecode definition of f oo () , wrap it in our own class and, (re)compile it. However, 



there is an issue here. What about other functions it may use, like shown in Figure 23 
with two methods calling each other? That would mean extracting those dependencies 
as well along with the method of interest. This won't scale very efficiently. Thus, 



alternate approaches include: to either inherit from the desired class as in Figure 19 



encapsulate this class instance, or attempt to wrap the entire class as done for the 



JAVA segment in Section 3.1.1.3 below. The former approach would imply having 
a class variable instance of the type of that class encapsulated into the wrapper. 
The latter approach was chosen as more feasible to implement, although it doesn't 
deal with user-defined classes and subclass and packages the .class or .Java file 
may require at the moment. Thus, the embed () acts in a way similar to #include in 
C/C++ or import in Java of a set of Java definitions to be used in a JLucid program. 
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Therefore, embed () has to be resolved at compile time. Similar technique may be 
taken towards other languages than Java at a later time. Lucid's syntax has to be 
extended to support embed (). 



3.1.1.3 The #JAVA and #JLUCID Code Segments 

This section explores ways of mixing Java and Lucid source code segments in a single 
text file and ways of dealing with such a merge. 



F 

where 

dimension d; 
F = foo(#d) ; 
where 

foo(i) = int fooCint i) { return i + 1; } 

end; 

end; 



Figure 20: Inline Java function declaration. 



An attempt to use Java's methods inline, such as in Figure |20] would be intuitive, 
but does not justify the effort spent on syntax analysis. Therefore, we take the inline 
definition out of the Lucid part, and make it a separate outer definition of the same 
method. Additionally, we explicitly mark the JLUCID and JAVA code segments to 



simplify pre-processing of the JLucid code as presented in Figure 21 



#JAVA 

int fooCint i) 
{ 

// Some i + PI 

return (int) (java. lang. Math.PI + i) ; 

} 

#JLUCID 
F 

where 

dimension d; 
F = foo(#d); 

end; 



Figure 21: Java method declaration split out from the Lucid part. 



Given the Natural Numbers Problem (see |Paq99| ) in Figure 22 (replicated here 
for convenience), one could imagine the function definition for to be implemented 
in Java in two functions. To illustrate the point when two separate functions can call 
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each other in the JAVA segment or several JAVA segments. This modified JLucid code 
along with line numbers is shown in Figure 23 Since we allow one Java method to 
call another within, we have to wrap them both into the same class. 



N S.d 2 
where 

dimension d; 

N = if (#.d <= 0) then 42 else (N + 1) O.d (#.d - 1) fi; 

end; 



Figure 22: Natural numbers problem in plain GIPL. 

The JLucid code segments after "#JAVA" constructs will be grouped together by 
the compiler. For all definitions (functions, classes, variables) in these segments, their 
original location in the JLucid source recorded and statically put in the wrapper class. 
These definitions will end up in that wrapper class as well. 

It would be possible to have a class defined within a wrapper class or any other 
valid Java declaration; even a data member can be included. To summarize, the Java 
segments in the JLucid code are a body of a generated class that implements the 
ISequentialThread interface. 



1 


#JAVA 


2 






3 


int 


getN(int piDimension) 


4 


{ 




5 




if (piDimension <= 0) 


6 




return get42 () ; 


7 




else 


8 




return getN (piDimension - 1) + 1; 


9 


} 




10 






11 


int 


get42() 


12 


{ 




13 




return 42; 


14 


} 




15 






16 


#JLUCID 


17 






18 


N St 


i 2 


19 


where 


20 




dimension d; 


21 




N = getN(#d); 


22 


end 





Figure 23: Natural numbers problem with two Java methods calling each other. 



For the example in Figure 23 the parser would proceed as follows: 
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In the preprocessing step the source code is spht into two parts: the Java part 
and the Lucid part. For both parts original source's hne numbers and length of 
the definitions are recorded. 

Then they both are fed to the respective parsers. Java's part requires extra 
handling: the Java methods (one or more) defined in the code, have to be 
wrapped into a class and then JavaCompiler class that takes the Java portion 
of the source and feeds it to javac for syntactic and semantic analyses and 
byte code generation. They will become parts of a Sequential Thread, ST (see 



Section 3.3.3.1) definition fed to Workers (see Section 3.3.3.4) 



• The Lucid part is processed by the modified Lucid compiler (to include the 
syntactical modifications for arrays and embedO) and comes up with the main 
AST from that. 

• The Java STs are then linked into the main AST in place of nodes where the 
identifiers of these appear in the Lucid part of the program prior semantic 
analysis. 

Any method or other definition in the JAVA segment is wrapped into a class. The 
generated wrapper class will contain a Hashtable that maps method signature strings 
to their starting line in the original JLucid code plus the length of the definitions in 
lines of text they occupy statically generated and initialized. This is needed for the 
error reporting subsystem in case of syntax/semantic errors, report back correctly the 
line in the original JLucid program and not in the generated class. The class name is 
created automatically from the original program name, the machine name it's being 
compiled on, and a timestamp to guarantee enough uniqueness to the generated class' 
name to minimize confiict for multiple such generated classes. Thus, the JAVA segment 



in Figure 23 will transform into the generated class as in Figure 24 This is a short 



version; for more detailed one please refer to the Section B.3 In fact, after generating 
this class (and possibly compiling it) this situation can be viewed as a special case 
for embed 0, Section [3. l.L 2 or vice versa. Note, since we have no guarantee the Java 



methods are side-effects free in JLucid, their results are not cached in the warehouse. 

In |MPG05j we required foo() in the previous examples to be static. In fact, 
any method or other definition in the JAVA segment were to be transformed to become 
static while being wrapped into a class. For example, "int foo() {return 1;}" 
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public class <f ilenanie>_<inachine_naine>_<timestajnp> 
implements gipsy . interfaces . ISequentialThread •[ 

private OriginalSourceCodeInf o oOriginalSourceCodeInf o; 

// Inner class with original source code information 
public class OriginalSourceCodeInf o { 

// For debugging / monitoring; generated statically 

private String strOriginalSource = ... 

// Mapping to original source code position for error reporting 
private Hashtable oLineNumbers = new HashtableO; 

// Body is filled in by the preprocessor statically 
public OriginalSourceCodeInf o { 

Vector int_getN_int_piDimension = new VectorO ; 

// Start line and Length in lines 

int_getN_int_piDimension . add(new IntegerO)) ; 

int_getW_int_piDimension . add(new Integer(7)) ; 

oLineNumbers . put (" int getN(int piDimension) " , 
int_getN_int_piDimension) ; 

Vector int_get42 = new VectorO; 

int_get42.add(new Integer (11) ) ; 

int_get42.add(new Integer(4)); 

oLineNumbers. put ("int get42()", int_get42) ; 

} 

} 

// Constructor 

public <f ilename>_<machine_name>_<timestamp>() { 

oOriginalSourceCodeInf = new OriginalSourceCodeInf oO ; 

} 

/* 

* Implementation of the Sequent ialThread interface 
*/ 

// Body generated by the compiler 
public void run() { 

Payload oPayload = new PayloadO; 

oPayload.addC'd" , new Integer(42)) ; 

work (oPayload) ; 

} 

// Body generated by the compiler statically 
public WorkResult work (Payload poPayload) { 

WorkResult oWorkresult = new WorkResult () ; 

oWorkresult . add (getN (poPayload . getVaueOf ( " d" ) ) ) ; 

return oWorkResult; 

} 

/* 

* The below are generated off the source file nat2java.ipl 
*/ 

public static int getN(int piDimension) { 
if (piDimension <= 0) return get42(); 
else return getN (piDimension - 1) + 1; 

} 

public static int get42() { 
return 42; 

} 

} 



Figure 24: Generated Sequential Thread Class. 
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would become "public static int foo() {...}". We insisted on static declara- 
tions only because the sequential threads were not instantiated by the workers when 
executed. This restriction has been lifted during implementation as we instantiate 
and serialize the sequential thread class as needed. 

3.1.1.4 Is JLucid an Intensional Language? 

We treat JLucid as a separate specific intensional programming language (SIPL) 
rather than a part of a GIPSY program within existing Indexical Lucid implementa- 
tion. Here are some pros and cons of this approach and JLucid as a separate SIPL 
approach is the winner. Why extend it as a separate SIPL? 

• This would serve as an example on how to add other SlPLs. 

• This would allow us to keep the original Indexical Lucid clean and working. 

• This would allow functions with Java syntax to be used within a Lucid program 
as well as binary Java function calls of pre-compiled classes. 

• It can be extended to other languages as it turns out to be a successful approach. 
Why not to treat is as a separate SIPL? 

• We might want to have embedded Java (or other language) in any intensional 
language, not just Indexical Lucid. How to make that possible? 

• It is not truly an SIPL, but a hybrid. 



3.1.2 Syntax 

In JLucid, we extend the syntax of both GIPL and Indexical Lucid to support arrays. 
For example, it is useful to be able to evaluate several array elements under the same 
context. This is included by the last E rules of E[E, E] and [E,...,E] in both 
syntaxes. Arrays are useful to manipulate a collection Lucid streams under the same 
context. JLucid arrays are mapped to Java arrays on the element-by-element basis 
with the appropriate element type matching and may only correspond to arrays of 
primitive types in Java. The syntax also includes the embed () extension to allow 
including external Java code. The JLucid syntax extensions to GIPL and Indexical 



Lucid are presented in Figure 25 and Figure 26 
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id 

E(E, . . . ,E) 

if E then E else E fi 
# E 
E a E 

E where Q end; 

[E:E, . . . ,E:E] 

embed (URI, METHOD, E, E, 

E[E, . . . ,E] 

[E,...,E] 

dimension id,..., id; 
id = E; 

id(id, . . . ,id) = E; 
QQ 



Figure 25: JLucid Extension to GIPL Syntax 



bin-op 
un-op 



id 

E(E, . . . ,E) 

if E then E else E fi 
# E 

E E E 
E where Q end; 
E bin-op E 
un-op E 

embed (URI, METHOD, E, E, ... 
E[E, . . . ,E] 

[E E] 

= dimension id,..., id; 
id = E; 

id. id, . . . ,id(ld, . . . ,id) = E; 
QQ 

= fby I upon I asa I wvr 
= first I next I prev 



Figure 26: JLucid Extension to Indexical Lucid Syntax 



3.1.3 Semantics 



The JLucid extension to the operational semantics of Lucid (see Section 2.2.2.8 



on 



page 15) is defined in Figure 27 As in the original Lucid semantics, each type of 



identifier can only be used in the appropriate situations. Notation: 

• freefun, ff id, f f def mean a type of identifier is a hybrid free (i.e. object- 
free) function freefun, where f f id is its identifier and f f def is its definition 
(body). 

• The EfHd rule defines JLucid's free functions. 



The JLucid T^JAVAfRd rule add free function definition to the definition envi- 
ronment. 
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#JAVAffld 



V,V\- E -.id V,!^ h El, . . . ,En : VI, . . . ,v„ 
V{id) = (freefun, ff id, ffdef) 
C.P l-<ffid(Di,. .. V 

V,V^ E{Ei,...,E,,) -.v 

ffdef = frttype f±±d{fargtypei farg^g^ ,fargtypen farg^g^) 
©."Ph ffdef : I?t[ff id (freefun, ffid, ffdef)], "P 



Figure 27: Additional basic semantic rules to support JLucid 

3.2 Objective Lucid: JLucid with Java Objects 
3.2.1 Rationale 

Objective Lucid is a direct extension of JLucid. The original syntax of Indexical Lucid 
(and also for JLucid and GIPL) is augmented to support a so-called dot-notation. This 
allows Lucid to manipulate grouped data by using object's methods. In fact, the idea 
is similar to manipulating arrays in JLucid. The difference with the arrays is that 
they are manipulated as a collection of ordered data of elements of the same type, to 
be evaluated in the same context. However, an object that varies in some dimension 
implies that all its members, possibly of different types, also potentially vary along 
this dimension, but across objects, i.e. the objects themselves are not intensional. An 
object can be thought of as a heterogeneous collection of different types of members, 
which you can access individually using their name, whereas arrays can be thought 
of as a homogeneous collection of members that can be accesses individually using 
their index. 

Just like JLucid |MPG05j . Objective Lucid is being developed as a separate specific 
intensional programming language (SIPL) within the GIPSY for the same reasons: 
keeping the other implementations undisturbed and working while experimenting on 
this particular implementation. 



3.2.1.1 Pseudo-Objectivism in JLucid 

A pseudo-object-oriented approach is already present in JLucid. The program pre- 



sented in Figure |28] gives an example of a Java function returning an object of type 
Integer. In JLucid we are not able to manipulate this object directly in intensional 
programming as Java does, though we can provide methods, such as g() to access 
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properties of a particular Java object from within JLucid. However, that reduces 
legacy Java code reusability by forcing the programmer to add such functions in his 



code to be able to use it in the GIPSY. Another example in Figure [29] shows how one 
can make use of objects in JLucid by providing pseudo-free Java accessors similar to 
getComputedBar in the example. They are pseudo-free because they don't appear 
as a part of any Java class to a JLucid programmer explicitly, but internally they get 
wrapped into a class when the code is compiled. In Objective Lucid such explicit 
workarounds are not necessary anymore, but this gives us some ideas about how to 
actually implement some features of Objective Lucid in practice, i.e., the compiler 
can generate a number of pseudo-free accessors to object's members and use JLucid's 
implementation of Java functions internally. 

3.2.1.2 Stream of Objects 

An interesting question could be to ask: "What is an object stream?" Is it that the 
members of this object vary in the same dimension(s) or they can have "substreams" ? 
In Objective Lucid we answer this as decomposing public object's data members into 
primitive types and varying them or in simplified manner we employ object's effectors. 
Thus, when there is a demand say for the object's state (data members) at some time 
t, there will have to be generated demands for all of t between [0,t] where at time 
an instance of the object is created. Therefore, the object state changes in the 
[0, t] interval represent the object stream in the context of this thesis. There are two 
possible outcomes of this evaluation: either a portion of object's state is altered by 
an intensional program or the entire object. In the former case. Lucid only accesses 
some object's members via the dot-notation in the intensional manner, whereas in 
the latter case all the members of an object are altered in the intensional context 



implicitly. The examples presented in Figure [301 Figure 74, page 153 and Figure 76 



page 156 work on portions of an object, whereas the examples in Section 4.1.3.6 



page 109 work on all the members of an object at the same time. 
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#JAVA 

Integer f() 
{ 

return new Integer("1234") ; 

} 

int gO 
{ 

return f (). int Value ; 

} 

#JLUCID 

A 

where 

A = gO; 

end; 



Figure 28: Pseudo-objectivism in JLucid. 



#JAVA 

class Foo 
{ 

private int bar; 

public FooO 
{ 

bar = (int) (Math. random () * Integer . MAX.VALUE) ; 

} 

public int getBarO 
{ 

return bar; 

} 

public void computeMod(int piParam) 
{ 

bar = bar piParam; 

} 

} 

int getComputedBar (int piParam) 
{ 

Foo oFoo = new FooO; 

oFoo . computeMod(piParam) ; 

System. out. printlnC'bar = " + bar); 

return oFoo.getBarO ; 

} 

#JLUCID 

Bar 
where 

Bar = getComputedBar (5) ; 

end; 



Figure 29: Using pseudo-free Java functions to access object properties in JLucid. 
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3.2.1.3 Pure Intensional Object-Oriented Programming 

Objective Lucid lias presented a way for Lucid programs to use Java objects. Tliis 
may seem rather restrictive and may look like a workaround (though practical!). 
An interesting concept would be to extend the Lucid language itself to create and 
manipulate pure Lucid objects, not Java objects. This will allow addressing issues like 
inheritance and polymorphism and other attributes of object-oriented programming 
and will solve the problem of matching Lucid and Java data types. This is not 
addressed in this work, but attempted to be solved in |WP05] . 



3.2.2 Syntax 

The parser is extended to support the <objectref > . <f eature> dot-notation for the 
Lucid part of reference data types. The semantic analysis is augmented to accommo- 
date objects and user-defined data types. In doing so. Lucid is able to manipulate 
Java objects as well as access public variables and methods of these objects. An 



example is shown in Figure 30 This example manipulates a simple object E by eval- 
uating its state at some time "2" . The program begins with the construction of the 
object with f 1() (or one could call the object constructor directly), and then the rest 
of the expressions access public members x and f oo() of the object during expression 
evaluation. 



The Objective Lucid syntax is in Figure 31 It is a direct extension of the JLucid 



syntax in Figure 26 to support the dot-notation. Essentially, the extension is the 
E . id productions. Any E on the left-hand-side can evaluate to an object type, but 
the right-hand-side is always an identifier (Java class' data member or method). 



3.2.3 Semantics 

To support these extensions to JLucid, the Semantic Analyzer of JLucid requires more 
non-trivial changes than the syntax analysis and the dot-notation implementation 
due to arbitrary object data types. In order to perform type checks and apply the 
semantic rules of Lucid, we place the object data types into the definition environment 
T>, which is in fact a semantic equivalent to the data dictionary part of the GEER. 
This is partly solved by using the pseudo-free Java functions, which de-objectify the 
object members, but in order to be able to do so, we need to have the object types 
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#JAVA 

class ClassXB 
{ 

public int x; 
public float b; 

public ClassXB 
{ 

X = 0; b = 1.2; 

} 

public int fooCint a, float c) 
{ 

return x = (int) (x * a + b * c) ; 

} 

ClassXB addxCint b) 
{ 

X += b; 
return this; 

} 

} 

ClassXB flO 
{ 

return new ClassXBO; 

} 

#OBJECTIVELUCID 
/* 

* The result of this program should be the object E 

* to be evaluated at time dimension 2 with its 'x' 

* member modified accordingly. 
*/ 

E atime 2 
where 

dimension time; 

E = fi() fby.time A; 

A = E.addx(B) ; 

B = E.fooCA etime C, A) + 3; 
C = E.x * 2; 

end; 



Figure 30: Objective Lucid example. 
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bin-op 
un-op 



id 

E(E, . . . ,E) 

if E then E else E fi 
# E 

E a E E 
E where Q end; 
E bin-op E 
un-op E 

embed (URI, METHOD, E, E, ... 
E[E, . . . ,E] 

[E E] 

E.id 

E.ld(E, . . . ,E) 
dimension id,..., id; 
id = E; 
E.id = E; 

id. id, . . . ,id(id, . . . ,id) = E; 
QQ 

fby I upon I asa I wvr 
first I next I prev 



Figure 31: Objective Lucid Syntax 

in the definition environment. The corresponding operational semantic rules from 
|Paq99| can be extended as follows. 

The Objective Lucid extension to the operational semantics of Lucid is defined 



in Figure 32 As in the original Lucid semantics, each type of identifier can only be 
used in the appropriate situations. Notation: 

• class, cid, cdef means it is a Class type of identifier with name cid and a 
definition cdef . 

• classv, cid.cvid, vdef means that the variable is a member variable of a 
class classv with identifier cid.cvid given the variable definition vdef within 
the class. 

• <cid.cvid> means object-member reference within an intensional program. 

• classf , cid.cfid, f def means that the function is a member function of a 
class classf with identifier cid.cfid given the variable definition fdef within 
the class. 

• <cid.cf id(f 1, . . . ,Vn)> represents a object-function call within an intensional 
program with actual parameters. 

• freefun, ff id, f f def mean a type of identifier is a hybrid free (i.e. object- 
free) function freefun, where f f id is its identifier and f f def is its definition 
(body). 



58 



• By cdef = Class cid {. . .} we declare a class definition. A class can contain 
member variable vdef and member functions definitions f def . 



The rules: 



• The Ec_vid rule defines an object member variable for an expression for the dot- 
notation. It is independent from the language in which wc define and express 
our objects. The rule says that under some context given two expressions E 
and E' that evaluate to a class-type identifier id and a variable type identifier 
id' respectively and if the two together via a dot-notation represent an object- 
data-member reference, then the expression E.E' evaluates to a value v. 

• Member function calls are resolved by the Ec-fct rule. Similarly to the Ec-vid 
rule, it defines that given two expressions E and E' under some context that 
evaluate to a class-type identifier id and a member function type identifier 
id' and a set of intcnsional expressions Ei, . . . , En evaluates to some values 
vi, . . . ,Vn and the two identifiers via a dot-notation represent a member function 
call with parameters Vi, . . . , then we say the expression E.E'{Ei, . . . , E2) is 
a member function call that under the same context evaluates to some value 
V, i.e. the function always returns a value. Here we see why it is necessary 
for Lucid to map a void data type to implicit Boolean true. This choice may 
seem a bit arbitrary (for example, one could pick an integer 1), but aside from 
practicality aspect the mere choice of true may signify a successful termination 
of a method. 

• The EfHd rule defines JLucid's free functions. The rule is a simpler version of 
Ec-fct with no class type identifiers present. 

• The #JAVAobjid rule places class definition into the definition environment. 

• The #JAVAobvjid ^-nd #JAVAobjfid rules add public Java object member 
variable and function identifiers along with their definitions to the definition 
environment. 

• The JLucid TTtJAVAfHd rule add free function definition to the definition envi- 
ronment. 
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Ec-vid 
Ec-fct 
Effld 

#JAVAobjid 
#JAVAobjvid 
#JAVAobjfld 
#JAVAffld 



©(id) = (class, cid, cdef) V (id' ) = (classv , cid.cvid, vdef) 
13, "P h<cid.cvid>: v 

V,V h E.E' : V 

V,'P\-E:id V,'Pi- E' -.id' V,V h Ei, . . . , E„ : vi, . . . ,Vn 
©(id) = (class, cid, cdef ) ©(id') = (classf , cid.cfid, fdef ) 
VjV l-<cid.cf id('yi, . . . ,v„)>: v 
V,Vh E.E'{Ei,...,En) -.v 

V,V h E -.id V,P \- El,. . . ,En : v-L,. ..,v„ 
T>{id) = (freefun, ff id, ffdef) 
V,r l-<ffid(i>i, . . . ,!>„)>: V 
V,r\- E{Ei, ...,£„): D 

cdef = Class cid {. . .} 
V,'P\- cdef : ©t[cid i-^ (class, cid, cdef)], V 

cdef = Class cid {. . . vdef . . .} vdef = public type vid; 
©.P h cdef : ©t[cid.vid 1-^ (classv, cid. vid, vdef)],^ 

cdef = Class cid {. . . fdef . . .} fdef = public frttype iid{fargtypei fargid^ , 



. ,fargtype„ fargu^) 



©.P h cdef : ©t[cid.fid 1-^ (classf , cid. fid, fdef )],P 
ffdef = frttype ffid{fargtypei fargid^^ Jargtypen fargu^) 



V,V\- ffdef : ©t[ff id i-» (freefun, ffid, ffdef)], P 



Figure 32: Additional basic semantic rules to support Objective Lucid 

3.3 General Imperative Compiler Framework 
3.3.1 Rationale 

Having to deal with JLucid, Objective Lucid, and Java and a future likely possibility 
to include other than Java imperative languages into intensional ones prompted inven- 
tion of a general mechanism to handle that and simplify addition of new languages 
into the GIPSY for research and experiments. This generalization touches several 
critical aspects exposed by the JLucid and Objective Lucid languages involving such 
a hybrid programming model. Thus, a core redesign of the GIPC was necessary to 
enable this feature. The General Imperative Compiler Framework (GICF) addresses 
the generalization issues (split among this Methodology and Design and Implemen- 
tation chapters) for the imperative compilers and suggests later development of a 
similar framework for the intensional languages. 

The core areas in the hybrid compilation process affect the way an intensional lan- 
guage program (which now syntactically allows having any number of code segments 
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written in one or more imperative languages) is compiled. This kind of program has 
to be preprocessed first to extract the code segments to be compiled by the appropri- 
ate language compilers and at the same time maintains syntactic and semantic links 
between the parts of a hybrid program. This influences the general intensional com- 
piler instrumentation, such as generation of sequential threads and communication 
procedures, function elimination, GIPL-to-SIPL translation, semantic analysis, and 
linking (and later interpreting/executing) of a GIPSY program. 

Requirements for any such a framework like GICF imply at least the following 
considerations: 

• having a number of compiler interfaces known to the system that any concrete 
compiler implements, 

• ability to pick such compilers at runtime based on a hybrid program being 
compiled, 

• have a generalized AST that is capable of holding intensional and imperative 
nodes, 

• have the semantic analyzer understand possible data types that any language 
may expose (which is a very challenging goal to do correctly), and deal with 
function elimination for the imperative parts of the AST, 

• preprocess by breaking down a hybrid GIPSY program's source code to be fed to 
the appropriate compilers gives us flexibility of allowing to include any impera- 
tive language we want, but complicates maintenance of semantic links between 
the intensional and imperative parts for later linking and semantic analysis. 
This necessitates development of the two other special segments that can de- 
clare in a uniform manner for GIPSY providing some meta information about 
embedded imperative sequential threads, like function and type identifiers, pa- 
rameter and return types for communication procedures, and user data types. 
Thus, for the former we need a function prototype declaration segment, that 
lists all free functions declared within imperative segments to be used by Lucid 
and the type declaration segment for the user-defined types possibly declared 
in those same imperative segments. The purpose of this meta-information is 
two-fold: it will help us maintaining the semantic links via a dictionary and 
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create so-called "imperative stubs". The former prompts the development of 



the GIPSY Type System (see Section 4.1.1.5, page 83) as understood by the 



Lucid language and its incarnation within the GIPSY to handle types in a more 
general manner. The latter stubs have to be produced in order for the inten- 
sional language compilers (that stay intact with the introduced framework) not 
to choke on "undefined" symbols that really were defined in the imperative 
parts, which an existing intensional compiler running in isolation fails to see. 

• After all involved compilers are finished doing compilation of their code seg- 
ments, they all produce a partial AST. For intensional compilers that means 
the main AST with the intensional and stub nodes. For imperative compilers it 
is the appropriate imperative AST for each sequential thread. The imperative 
AST, in fact, need not to be a real tree and may contain a single imperative 
node that would hold a payload of STs (compiled object or byte code), CPs, 
type information, and some meta-information (e.g. what language the STs and 
CPs are in and for which operating system and native compiler environment). 

• Then, the imperative stubs have to be replaced by the real imperative nodes at 
the linking stage before the semantic analysis. 

• Once the main tree is formed, the semantic analyzer would use the type system 
to verify type information of the intensional-imperative calls within taking into 
consideration imperative nodes when doing function elimination and producing 
the final "executable" tree, or Demand AST, or DAST, a component of the 
GEER. 

All this work is motivated by the desire to simplify the addition of new compilers 
into the GIPSY environment with minimal integration hassle. The follow up sections 
explore some of the issues about primary matching of the Java and GIPSY data 
types, followed by the definition of sequential threads and communication procedures 
in the GIPSY, and their Worker aggregator. While the below are sections that lay 
down a concrete example based on JLucid and Java, the discussion addressing the 
generalization of the design and implementation of these issues are presented in the 
chapter that follows with the actual sequence diagram showing implementation details 
of the above hybrid compilation process. 
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3.3.2 Matching Lucid and Java Data Types 

Allowing Lucid to call Java functions brings a new set of issues related to data types. 
Additional work is required on the semantic analyzer, especially when it comes to 
type checks between Lucid and Java parts of a JLucid program. This is pertinent 
when Lucid variables or expressions are used as parameters to Java functions and 
when a Java function returns a result to be assigned to a Lucid variable or used in 
an IP expression. The sets of types in both cases are not exactly the same. The 
basic set of Lucid data types as defined by Grogono |Gro02b] is int, bool, double, 
string, and dimension. Lucid's int is of the same size as Java's int, and so are 
double, boolean, and String. Lucid string and Java String are simply mapped 
to each other since internally we implement the former as the latter; thus, one can 
think of the Lucid string as a reference when evaluated in the intensional program. 
Based on this fact, the lengths of a Lucid string and Java String are the same. 
Java String is also an object in Java; however, at this point, a Lucid program has no 
direct access to any object properties. We also distinguish the float data type for 
single-precision floating point operations. The dimension index type is said to be an 
integer for the time being, but might become a float when higher precision of points 
in time, for example, will be in demand, or it could even be an enumerated type 
of unordered values (though float dimensions will introduce some very interesting 
problems). Therefore, we perform data type matching as presented in Table [l] The 
return and parameter types matching sets are not the same because of the size of the 
types. Additionally, we allow void Java return type which will always be matched 
to a Boolean expression true in Lucid as an expression has to always evaluate to 
something. 

The table does not reflect the fact that JLucid is able to manipulate arrays of 
values (streams), but these arrays are not Java arrays (Java's arrays are objects). In 



Objective Lucid (see Section 3.2), we also have Java object data types will also be 
manipulated by a Lucid program with the Lucid part being able to access object's 
properties and methods and have them as return types and arguments. As for now 
our types mapping and restrictions are as per Table [Tj 
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Table 1: Matching data types between Lucid and Java. 



Return Types of Java Methods 
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Parameter Types Used in Lucid 


Corresponding Java Types 


string 


String 


float 
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double 


double 


int, dimension 


int 


bool 


boolean 



3.3.3 Sequential Thread and Communication Procedure Gen- 
eration 

3.3.3.1 Java Sequential Threads 

Sequential threads are imperative functions that can be called in the Lucid part of 
a GIPSY program. The data elements of a Lucid program are integers and the like. 
Using them as such would result in a very inefficient computation due to the overhead 
in generation and propagation of demands. STs overcome this problem. The notion of 
sequential thread and granularization of data was introduced by the GLU (Granular 
LUcid system jJDQGllJDAQTj . 

Each GIPSY program potentially defines several Java methods that can be called 
by the Lucid part of the program. Each of these functions are coded in the Java part 
of the GIPSY program; thus, a sequential thread represents by itself a bit of work to 



compute split into one or more Java methods. They are compiled (see Figure 33 ) to 



Java byte code by the compiler (GIPC, Figure 10) and packed into one executable 



along with the Communication Procedures (CP) (see Section [3.3.3.2) needed for the 



communication between the generator and worker (Section 3.3.3.4, Figure 34). The 
notion of worker is thus very close to the notion of sequential threads, where a worker 
is basically the aggregation of the (potentially) several sequential threads that can 
be executed by a worker, along with the communications procedures needed for the 
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Figure 33: Hybrid GIPSY Program Compilation Process 



generator-worker communication. 

Notice that the Generator- Worker Architecture may well be extended so that the 
worker and the generator are fused into one; this is under review and is discussed in 
|Lu04] and in |VP05] . This gives us distributed generators as outlined in |Gro02b] . 
but as yet is only a topic for discussion. 



3.3.3.2 Java Communication Procedures 

The functional demands (i.e., demands that raise the need for a Java function call) 
are potentially computed by remote workers, upon demand by the generator. The 
demand is sent via the network by the generator to the worker, along with the data 
representing the parameters of this Java function call. Sending this data through the 
network requires the breaking of the data structure into packets transmissible via a 
network. This packing of the demand's input data is done by the Communication 
Procedures, along with some kind of remote procedure call to the worker using, for 
example, TCP/IP RPC. Once the function (the sequential thread) resolves, the worker 
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(Section 3.3.3.4) is responsible for sending back the result to the generator that called 
for this demand. That is also done by the CPs. 

The CPs are generated by the compiler (GIPC) using the first part of the GIPSY 
program: the definition of the data structures sent over the network (i.e., the pa- 
rameter and return types of the Java functions). The GIPC parses these Java data 
structures and translates them into an abstract syntax tree. This tree is then traversed 
by the CP generator, which generates byte code for the communication procedures, 
following the communication protocol that was selected. Serialization summarizes 
much of this and Java helps us do it. 

The CP generator has to be extremely flexible, as it has to be able to generate 
code that uses various kinds of communication schemes. In a nutshell, CPs determine 
the way a ST should be delivered to the computing host's worker depending on 
the communication environment. For the localhost, it is plain TLP (i.e., we create 



Java threads on a local machine) so NullCommunicationProcedure (Section B.2) is 
used. For distributed environment CPs wrap transport functions over Jini, DCOM+, 
CORBA, PVM, and RMI (see jCTiUVPOK] ) protocols. Both CP and ST interfaces 



are presented in Section 4.1.1.8 



3.3.3.3 C Sequential Threads and Communication Procedures with the 
JNI 

This is the methodology of how to extend the Java ST/CP generation concepts to 
C (and similarly can be done for C++) with the JNI |Ste05j introduced in Sec- 



tion 2.6.1.2 page 36 This approach was designed, but not implemented as of this 
writing; however, it may serve as a good head start on the implementation of the 
CCompiler in GICF. 

Much of the ST wrapper class generation code for C will be similar to that of 
Java. The main difference is the bodies of the sequential thread functions will not 
be present in the generated class as-is, but they will be declared as native with 
no Java implementation. The C code chunks will be saved to a . c file and the 
corresponding .h fill will be generated declaring all the needed prototypes with the 
javah tool provided with the standard distribution of the JDK. After that, we call an 
external C compiler to compile the C chunks into a shared library. Thus, the other 
modification to the generated wrapper class the CCompiler has to do, is to add a 
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static initializer with the System. loadLibrary () call for the newly compiled library 
with the C implementation of our ST(s). The generated ST class and the compiled 
mini-library can be stored together (e.g. the binary library file can be loaded into 
a byte array of the class and deserialized back when about to be executed) in the 
imperative node and later be communicated just like Java STs. A more sophisticated 
alternative is to do the compilation and dynamic loading after communication by the 
engine, but this can be a next step. 

As far as type matching concerned, we still can use the same mapping rules defined 



in Section 3.3.2 (and subsequently the TypeMap class of the JavaCompiler presented 
later on) because with the JNI with still work with Java and the JVM can do Java- 
to-native type translation to C or C++ for us, not only for primitive types, but also 
for arrays, objects, and strings. 

3.3.3.4 Worker Aggregator Definition in the Generator- Worker Architec- 
ture 



The GIPSY uses a generator- worker execution architecture as shown in Figure 34 
The GEER generated by the GIPC is interpreted (or executed) by the generator fol- 
lowing the eductive model of computation. The low-charge ripe sequential threads 
are evaluated locally by the generator. The higher-charge ripe sequential threads are 
evaluated on a remote worker. The generator consists of two systems: the Intensional 
Demand Propagator (IDP) and the Intensional Value Warehouse (IVW) |Tao04] . The 
IDP implements the demand generation and propagation mechanisms, and the IVW 
implements the warehouse. A set of semantic rules that outlines the theoretical as- 
pects of the distributed demand propagation mechanism has been defined in |Paq99] . 
The worker simply consists of a "Ripe Function Executor" (RFE), responsible for 
the computation of the ripe sequential threads as demanded by the generator. The 
sequential threads are compiled and can be either downloaded/uploaded dynamically 
by/to the remote workers. Better efficiency can be achieved by using a shared network 
file system. 

An example: a GIPSY screen saver would be a sample worker running when the 
an ordinary PC is going into an idle mode and normally launches ordinary dancing 
bears screensavers, it can actually run our downloaded worker instead and contribute 
to computation. When such a worker starts, it has to register it within a system 
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somehow (see |VP05] ). so that the generators are aware of its presence and can send 
demands to it. In the event of merging of semantics of a worker and a generator, such 
a Screensaver would also be able to generate demands and maintain a local warehouse. 



3.4 Summary 

This chapter presented methodology behind concrete implementations of the first 
two hybrid languages in the GIPSY - JLucid and Objective Lucid. Semantic rules 
were presented for free Java functions and Java objects to be included into the Lucid 
programs and evaluated by the eduction engine in the hybrid environment. Further- 
more, operational semantics of Objective Lucid is clearly defined and is compatible 
with the semantics of Lucid. The general requirements for the GICF, a tool simpli- 
fying imperative compiler management within GIPC, are introduced. The follow up 
chapter details the architectural and detailed designs and concrete implementation of 
the languages as well as General Intensional Compiler Framework and overall module 
integration and their interfaces. Some immediate benefits and limitations are outlined 
below. 
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3.4.1 Benefits 

• JLucid opens the door for STs and CPs and first hybrid programming paradigm 
in the GIPSY. 

• JLucid provides abihty to either write Java code alongside the Lucid code or 
embed existing one via embedC). 

• Objective Lucid introduces Java objects and their semantics in the GIPSY. 

• GICF generalizes the embed () mechanism to all languages in the GIPSY. 

• GICF promotes general type handhng in the GIPSY. 

• GICF promotes general compiler handling in the GIPSY. 

• GICF generahzes the notion of the STs and CPs for all compilers. 

3.4.2 Limitations 

• JLucid is limited only to GIPL-Java and Indexical Lucid- Java hybrids. 

• JLucid does not allow Java objects. 

• JLucid restricts the embed () mechanism only to itself and its derivative - Ob- 
jective Lucid. 

• Objective Lucid is primarily an experimental language to research on Java ob- 
jects in the intensional environment. 

• GICF addresses mostly the imperative compilers, but a similar approach can 
be applied to the intensional and functional ones. 
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Chapter 4 

Design and Implementation 



This chapter combines the architectural and detailed designs and integration of the 
modules contributed not only by the author of this thesis but also by the other GIPSY 
team members. Section |4.1 explores the GIPSY architecture and implementation 
of the major components and frameworks. Then, Section 4^ focuses on the user 
interface and external library interfaces. User interfaces, class and sequence diagrams 
are provided mostly following the top-down approach. For GIPSY Java packages, 
directory structure with description of each package, and .jar file packaging please 
refer to Appendix ICj 



4.1 Internal Design 

The GIPC framework redesign along with the realization of the two children frame- 
works of GICF and IPLCF are presented first followed by the design and implemen- 
tation of JLucid and Objective Lucid integrated into the new frameworks. 



4.1.1 General Intensional Programming Compiler Framework 

The GIPC Framework experienced several iterations of refinements as a result of 
this research. Two new frameworks emerged, namely General Imperative Compiler 
Framework (GICF) to handle all imperative languages within the GIPSY and, its 
counterpart Intensional Programming Languages Compiler Framework (IPLCF). 
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4.1.1.1 General Imperative Compiler Framework 

GLU |,TDA97[|JD96] . JLucid |MP(;n5] . and later Objective Lucid |MPn5bj prompted 
the development of a General Imperative Compiler Framework (GICF). The frame- 
work targets integration (embedding of) different imperative languages into GIPSY 
(see |RG05aj ) programs for portability and extensibility reasons. GLU promoted C 
and Fortran functions within; JLucid/ Objective Lucid promote embedded Java. Since 
GIPSY targets to unite all intensional paradigms in one research system, we try to 
be as general as possible and as compatible as possible and pragmatic at the same 
time. 

For example, if we want to be able to run GLU programs with minimum (if at 
all) modifications to the code base, GIPSY has to be extended somehow to support 
C- or Fortran-functions just like it does for Java. What if later on we would need to 
add C++, Perl, Python, shell scripts, or some other language for example? The need 
for a general "pluggable" framework arises to add imperative code segments within 
a GIPSY program. We could go even support multi-segment multi-language (with 
multiplicity of 3 or more languages) GIPSY programs. Two examples are presented 



in Figure M5\ and in Figure 36 



4.1.1.2 Generalization of a Concrete Implementation 



Thus, the JavaCompiler component (see Figure 33), part of GIPC, has to be gen- 
eralized, and the JavaCompiler itself be a concrete implementation of this gen- 
eralization. The generalization would express itself by having an abstract class 
ImperativeCompiler, the generic Preprocessor (vs. JLucidPreprocessor in Sec- 
tion 4.1.2) should be able to cope with all PLs and know what PLs are supported 
through enumerating them. Another thing the GICF buys us is an ability to have any 
supported imperative programming language embedded in any supported intensional 
programming language. Though this may seem impractical at the first glance, but 
the framework is designed such that a lot of syntax, semantics, and type mapping 
work is performed by the individual concrete compiler implementations and not by 
the generic machinery. The goal here is that as long as any given compiler within 
the framework conforms to the designed interface specification and produces the re- 
quired data structures, there should be least possible effort to enable such a compiler 
in GIPSY. Thus, the compilation process, semantic checks, linking, and execution at 
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#f uncdecl 

Integer f () ; 
void geeO ; 
void z() ; 

#typedecl 

Integer ; 

#JAVA 

Integer f() 
{ 

return new Integer("123") ; 

> 

#CPP 

#include <iostream> 

void geeO 
{ 

cout « "gee" << endl; 

} 

#PERL 

sub z() 
{ 

while «STDIH» 
{ 

s/W/; 
print ; 

} 

} 

#OBJECTIVELUCID 

A a.d 5 
where 

dimension d; 

A = B fby.d (A - 1); 

B = C fby.d (B + f (). int Value ()) ; 

C = z() && geeO ; 

end; 



Figure 35: Example of a hybrid GIPSY program. 
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/** 

* Language-mix GIPSY program. 
* 

* $Id: language-mix. ipl.v 1.5 2005/04/25 00:16:30 mokhov Exp $ 

* $Revision: 1.5 $ 

* $Date: 2005/04/25 00:16:30 $ 
* 

* aauthor Serguei Mokhov 
*/ 

#typedecl 
myclass ; 
#funcdecl 

myclass foo (int .double) ; 

float bar (int , int) : "ftp://newton.cs . concordia. ca/cool . class" :baz 
int flO; 

#JAVA 

myclass fooCint a, double b) 
{ 

return new myclass (new Integer ( (int) (b + a))); 

} 

class myclass 
{ 

public myclass (Integer a) 
{ 

System. out . pr int In ( a) ; 

} 

} 

#CPP 

#include <iostream> 

int fl(void) 
{ 

cout « "hello"; 
return 0; 

} 

#OBJECTIVELUCID 

A + bar(B, C) 
where 

A = foo(B, C). int Value (); 
B = flO; 
C = 2.0; 

end; 
/* 

* in theory we could write more than one intensional chunk, 

* then those chunks would evaluate as separate possibly 

* totally independent expressions in parallel that happened 

* to use the same set of imperative functions . 
*/ 

// EOF 



Figure 36: Another example of a hybrid GIPSY program. 
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the meta level of implementation of the GIPC and GEE can be reasonably generalized 
without loss of practicality as we shall see. With this great deal of flexibility, we have 
several issues: 

• Binary portability of compiled languages, such as C/C++ on a different host 
(this problem theoretically does not exist for Java). 

• Though some languages, such as Perl, Python, shell scripts, are interpreted, a 
version mismatch may happen. 

• A compiler for interpreted languages other than Java would be rather simple 
because should we want to pass the ST code to a remote host, all we need is 
to pass the source itself. Of course, in both compiled and interpreted variant 
there is a large potential of security vulnerability exploits (e.g. with malicious 
code injection), which will have to be dealt with as a part of the future work. 
As of this writing, there are no embedded checks in GIPSY for that; instead a 
guide of a sandboxed installation of GIPSY will be provided when the system 
is released. 

• Another important issue is having imperative PL nodes in the AST. The issue 
is in what such nodes should contain in order for them to be linked back into 
the main AST, how to perform semantic analysis of the hybrid code based on 
the contents of such nodes, and GEE should go about executing this code. 

• Various languages define their own set of types and typing rules, gluing them 
all together is a very difficult task for semantic analysis and type inference. 

The follow up sections clarify and address most of these issues. 



4.1.1.3 Resolving Generalization Issues and Binary Compatibility 



In order to fully support GICF, the original GIPC framework in Figure 37 (discussed 
in detail by Wu and Paquet in |PGW04j ) has to be altered in the following way: the 
Preprocessor has to be added on top of all the front-end modules, and new links drawn 



between the Preprocessor and the other modules Figure 38 This also changes the 
data structures flow between the components. For the unaware reader, what follows 
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is the brief description of the layers, components, and abbreviations of the conceptual 
design present in Figure |38] 

The front-end and back-end layers are the two bottom ones represent the main 
machinery of the GIPC. The front-end compilers and parsers are responsible for pars- 
ing, producing initial syntax trees, STs, and CPs. At this layer, the main abstract 
syntax tree AST is always compliant to the one of Generic Intensional Programming 
Language (GIPL). If the source code program was written in some specific intensional 
programming language (SIPL, e.g. Indexical Lucid or Tensor Lucid), its AST has to 
be translated first into GIPL. Both, GIPL and SIPL type components may translate 
a Lucid dialect source code into a data flow (DFG) graph language and back; hence, 
there is a variety of the DFG translators. Next, the other two types of conceptual 
components at the front-end layer are the data type (DT) and the sequential thread 
(ST) front-ends. These correspond to the imperative language compilers and their 
modules in the implementation. The DT front-end is responsible for analyzing data- 
type definitions in the ST code and producing native (i.e. compiled) representation of 
communication procedures (NPCs). The ST front-end is responsible for compilation 
an ST code and producing some equivalent of the native compiled code (NST) as the 
end result. 

The GIPC back-end layer performs finalization of a GIPSY program compilation 
by doing semantic analysis and eliminating Lucid functions and producing the demand 
AST (DAST) along with linking in the generated STs and CPs from the imperative 
side. The GEER generator then produces the final linked version of a GIPSY program 
as a resource usable by the GEE (GEER). 

The first two layers are meta-level layers that prepare information for the front- 
end and back-end layers. The second layer is the GIPC Preprocessor layer discussed 
in depth through the rest of this chapter. The top level has to do with some language 
specification processing and creating corresponding parsers and data structures for 
the front-end layer. SIPL and GIPL front-end generators have to do with the fact 
that our SIPL and GIPL parsers are generated out of a source grammar specification 
by javacc. Thus, a GIPL specification corresponds to the GIPL grammar in the 
GIPL. j jt file and the GIPL spec processor is the javacc tool. The DT and ST front- 
end generators exist for the same idea as the GIPL and SIPL ones do. However, in 
the current implementation they are not present either because they are hand-written 
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Figure 37: Original Framework for the General Intensional Programming Compiler 
in the GIPSY 

or we rely on the external compiler tools (e.g. javac to compile Java STs) to do the 
processing for us. The design however implies that these components may eventually 
be converted to the genuine imperative compilers within GIPSY giving greater control 
and flexibility over the imperative parts than relying on external tools. Therefore, we 
may acquire a Java, jjt one day, for example, and generate a Java parser out of it. 



Format Tag To address some binary compatibility issues we invent a notion of 
a format tag attached to the STs and CPs. The format tag's purpose is to include 
meta-information about STs and CPs such that it includes the programming language, 
the object code format, the operating system, compiler, and their versions. This is 
important if we are sending platform-dependent compiled code, such as that of C or 
C++ from one host to another with different architectural platforms. The FormatTag 



API is in Figure 39 



We implement format specifications as a hashtable. We also predefine some com- 
mon format tags, such as JAVA, for conveniences as most frequently used. The class 
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Figure 39: The FormatTag API. 



overrides toStringO and equals () of Object to define that the two format tags are 
only equal if the string representation of all their specifications are identical. 

Sending Source Code Text Not all non-intensional languages require compila- 
tion, e.g. Perl, Python, etc. These can be sent over as plain source code text; thus, the 
format tag will indicate the fact. We can go even further with this and send any lan- 
guage as plain text and compile it on the target host instead prior invocation. For the 
task of the source code inclusion we reserved the SequentialThreadSourceGenerator. 
Of course, this won't work for embed-included binary code via a URI parameter be- 
cause that code was already compiled by someone else on some specific platform. As 
far as current implementation concerned, the generated ST class does always contain 
the source code of STs from the GIPSY program code segments, but it is unused by 
the GEE except for debugging as of this writing. 

Dictionary The Preprocessor's dictionary will initially be constructed based on 
the #funcdecl and #typedecl program segments. The dictionary will serve as an 
input to three other components: the NST generator (for error reporting and point- 
ers to the nodes in the AST and the compiled code), to the NCP generator (to 
analyze the data structures used by STs and generate CPs accordingly), and to the 
semantic analyzer, to perform data type matching between the intensional and im- 
perative parts. Both NCP and NST generators work under the command of some 
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imperative language compiler and are referred to as SequentialThreadGenerator 
and CommunicationProcedureGenerator in their most general forms, which are sub- 
classed by a concrete language implementation. 



4.1.1.4 GIPC Preprocessor 

The Preprocessor is something that is invoked first by the GIPC on incoming GIPSY 
program's source code stream. The Preprocessor's job is to do preliminary program 
analysis, processing, and splitting into chunks. Since a GIPSY program is a hybrid 
program consisting of different languages in one source file, there ought to be an 
interface between all these chunks. Thus, the Preprocessor after initial parsing and 
producing the initial parse tree, constructs a preliminary dictionary of symbols used 
throughout the program. This is important for type matching and semantic analysis 
later on. The Preprocessor then splits the code segments of the GIPSY program 
into chunks preparing them to be fed to the respective concrete compilers for those 
chunks. The chunks are represented through the CodeSegment class that the GIPC 



collects. The corresponding class diagram of is in Figure 40 

The Preprocessor can also be told to report certain code segments are invalid at 
the preprocessing stage rather delaying the error until the compiler discovery stage 
through the addlnvalidSegmentNameO and addValidSegmentNameO methods and 
maintaining internal vector of the strings with invalid segment names. This fea- 
ture is for example used in Preprocessor's extensions of JLucidPreprocessor and 
ObjectiveLucidPreprocessor later on that filter out code segments that do not 
belong to the languages. The filtering logic works like this: 

• if no valid and invalid segments are specified, all segments are accepted as valid 
at the preprocessing stage. This is the default for general GIPC work. 

• if some invalid and no valid segments are specified, the Preprocessor will error 
out on the invalid segments 

• if only valid segments are specified, everything else will be treated as invalid 

• if both valid and invalid segments are present; the invalid set segments are 
ignored and everything that it is not mentioned in the valid set is said to be 
invalid. 
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Preprocessor 

Cfrom Preprocesing) 



■j^oCodeSegments : Vector = new Vector Q 
■j^oValidSegmentNames : Vector^ new Vector Q 
■j^olnvalidSegmentNames : Vector = new Vector Q 
"S^olmperativeStubs : Hashtable = new Hashtable Q 



*PreprocessorO 
^preprocessQ 
^setSourceStreamQ 
^getDicti onaryO 
*getPreprocessorASTRootO 
^splitCodeSegmentsQ 
^producelmperativeStubsO 
*g et I mp e rat ive St u b s 
*getCodeSegment30 
*addValidSegmentNameO 
^addlnvalidSegmentNameQ 



CodeSegment 

^rom Preprocesing) 



^strLanguage : String = null 
^strCode : Stnng = null 



^CodeSegmentQ 
^getLanguageNameQ 
^getSourceCodeO 
♦toStringO 

^getSourceCodeStreamQ 



o 

PreprocessorParserTreeConstants 

(from Preprocessing) 



PreprocessorParser 

(f ro m Preprocesin £ 



Dictionary 

(from storage) 



^DictionatyO 



GfPSYType 



All but Preprocessor, t\ 
Dictionary, and 
CodeSegment are 
generated by javacc. 



♦SimpleNodeQ 
■♦jtOpenO 
*|tClose() 
^jtSetParentO 
^jtGetParentO 
^jtAddChildO 
^jtGetChildO 
*jtGetNLmChildren() 
♦toStringO 
♦toStringO 
^dumpO 
*setLej!emeO 
*getLej!emeO 
♦SimpleNodeO 



#(jtree 




JJTPreprocessorParserState 

Cfrom Preprocessing^ 



Anodes ; Stack 
^marks : Stack 
^sp : int 
^mk : int 

^node created : boolean 



■i^JJTP neprocesso rPars erSt ateQ 
■i^nodeCreatedO 
■i^reset _ 
■i^rootNodeO 
■i^pLshNo deQ 
T^popNodeQ 
T^peekNo deQ 
T^nodeAfityO 
T^clearNo deScope() 
T^openNo deScope() 
T^cioseNodeScopeO 
■^closeNodeScopeQ 



PreprocessorParserTokenManager 

(^frorn Preprocessing^ 



Prep 



rocessorParserConstants 

(from Preprocessing) 



Figure 40: The GIPC Preprocessor. 



GIPSY Program Segments Here we define four basic types of segments to be 
used in a GIPSY program. Tliese are: 

• #f uncdecl program segment declares function prototypes of imperative-language 
functions defined later or externally from this program to be used by the inten- 
sional language part. These prototypes are syntactically universal for all GIPSY 
programs and need not resemble the actual function definitions they describe 
in their particular programming language. 

• #typedecl segment lists all user-defined data types that can potentially be used 
by the intensional part; usually objects. These are the types that do not appear 
in the matching table in Table [T] 

• #<IMPERATIVELANG> segment declares that this is a code segment written in 
whatever IMPERATIVELANG may be, for example #JAVA for Java, #CPP for 
C++, #PERL for Perl, #PYTHON for Python, etc. 
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• #<INTENSIONALLANG> segment declares that this is a code segment written in 
whatever INTENSIONALLANG may be, for example #GIPL, #INDEXICALLUCID, 
#JLUCID, #OBJECTIVELUCID, #TENSORLUCID, #ONY)Q etc. as understood by the 
GIPSY. 



Preprocessor Grammar The initial grammar for the Preprocessor to be able 



to parse a GIPSY program is shown in Figure |4T} After having parsed a program, 
we have a Preprocessor AST (PAST) that will be used further by the compilation 
process in the GIPC and its submodules. The grammar and the framework were 
designed in such a way so all the previous neat features of JLucid |MP05b] / Objective 
Lucid |MP05bj still be present, such as embed () and are accessible to other dialects. 
In the GICF, we generalize our function prototype declaration to be able to include 
external code of any imperative language. 

The lexical elements, such as LETTER, LANGDATA, DIGIT, CAPLETTER, and 
*LITERALs are not listed for brevity as they are merely standard and self-explanatory 
lexical tokens except probably LANGDATA - this is character data allowing any 
character sequence within except LANGID that serves as a terminator of a code 
segment chunk. 

Notice, the grammar is not bound to our current set of supported intensional and 
imperative languages. Rather, the GIPC attempts to look up appropriate compiler 
for each code segment automagically using LANGID for mapping at run-time. The 
JavaCC version of the grammar can be found the PreprocessorParser . j jt file. 

The grammar has been amended from what was published in |MP05aj to include 
LANGID in the EMBED production, the immutable keyword and arrays subscript 
operator [] in the PSTART production. LANGID in EMBED is needed to be able to 
pick the appropriate compiler for the included code as it may be written in any 
imperative language. The immutable keyword is needed to allow a programmer 
to assert that certain STs are immutable meaning given the same parameters they 
always return the same result, and, therefore, their result can be safely cached in the 
warehouse as such functions are declared side-effects free (e.g. as the get42() method 



in Figure 23, page 48 can be marked as immutable). This marking of methods will 
allow more efficient caching of the ST results of STs known not to have side effects 



^See |Gro04| for details on the Onyx language. 
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<GIPSY> 




<DECLARATIONS> <CODESEGMENTS> 


<DECLARATIONS> 


1 
1 

1 


<FUNCDECLS> <DECLARATIONS> 

e 


<FUNCDECLS> 




#funcdecl <PROTOTYPES> 


<TYPEDECLS> 


::= 


#typedecl <TYPES> 


<PROTQTYPES> 


"l 


<PROTOTYPE> ; <PROTOTYPES> 

e 








<PSTART> 




[ inmutable ] <TYPE> [ [] ] <ID> ( <TYPELIST> ) 


<EMBED> 


-r 


e 

: <LANGID> : <URI> 


<^TYPES^ 




e 


<TYPELIST> 


I 


<TYPE> [ [] ] 

<TYPE> [ [] 1 , <TYPELIST> 

e 


<CODESEGMENT> 




<LANGDATA> <LANGID> 
<LANGDATA> <EOF> 


<CODESEGMENTS> 




<CODESEGMENT> <CODESEGMEKTS> 

e 


<URI> 




<CHARACTERLITERAL> 
<STRIHGLITERAL> 


<ID> 




<LETTER> (<LETTER> | <DIGIT>)* 


<LANGID> 




#<CAPLETTER> (<CAPLETTER>)* 


<TYPE> 




<ID> 
int 

double 

bool 

float 

char 

string 

void 



Figure 41: Preprocessor Grammar for a GIPSY program. 
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and has to be explicitly set by the programmer. If the programmer by mistake marks 
a method with side effects as immutable, then a program may exhibit erroneous 
execution at run-time by returning a possibly incorrect value from the warehouse. 
There is no way to automatically discover immutability of STs in GIPSY at this time 
(it may only be possible when genuine imperative compilers are implemented). The 
array subscript operator [] has been added to PSTART and TYPELIST productions 
to allow GIPSY arrays (as a generalization of JLucid arrays) that are composed of 
the elements of GIPSY types. The concrete imperative compilers implementing the 
mapping (if possible) will have to do appropriate conversions from the native arrays 
to GIPSY arrays. 



4.1.1.5 GIPSY Type System 

While the main language of GIPSY, Lucid, is polymorphic and does not have explicit 
types, co-existing with other languages necessitates definition of GIPSY types and 



their mapping to a particular language being embedded. Figure 42 presents the design 
aspects of the GIPSY Type System. 

Each class is prefixed with GIPSY to avoid possible confusion with similar def- 
initions in the java.lang package. The GIPSYVoid type always evaluates to the 



Boolean true, as described earlier in Section 3.3.2 The other types wrap around the 
corresponding Java object wrapper classes for the primitive types, such as Integer, 
Float, etc. Every class keeps a lexeme (a lexical representation) of the correspond- 
ing type in a GIPSY program and overrides toStringO to show the lexeme and 
the contained value. These types are extensively used by the Preprocessor, imper- 
ative and intensional (for constants) compilers, the SequentialThreadGenerator, 
CommunicationProcedureGenerator, SemanticAnalyzer for the general type of GIPSY 
program processing, and by the GEE Executor. 

The other special types that have been created are either experimental or do not 
correspond to a wrapper of a primitive type. GIPSYIdentif ier type case corresponds 
to a declaration of some sort of an identifier in a GIPSY program to be put into the 
dictionary, be it a variable or a function name with the reference to their definition. 
This is an experimental type and may be removed in the future. Constants and 
conditionals may be anonymous and thereby not have a corresponding identifier. 
GIPSYEmbed is another special transitional type that encapsulates embedded code via 
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GIPSYInteger 



li^QlntegerValue : Integer 



♦GIFSYIntegerO 

*GIPSYIntegerO 

♦eiPSYIntegerO 

♦getValueO 

*toStringO 

*getErclosedTvpeOjectO 



GIPSYDouble 
("■"■Una! 



i>oDoublGValue : Double 



♦GIPSYDoybleQ 

♦eiPSYDoubleO 

*toStringO 

♦GIPSYDoyblaQ 

*getEnclosedTypeOjectO 

♦gelValueO 



GIPSYIdentlfer 



i^strldentifierValue . String 



♦GIPSYIdentifierO 
♦GIPSYIdentifierO 
♦getValueO 
♦toStringO 

*getEnclosedTypeOjectO 



GIPSYOperator 



♦GIPSYOparalnrO 
♦GIPSYOperalorO 



GIPSYCharacler 



^oCharacterValuG Charactet 



♦GIPSYCbaracterO 

♦GIPSYCharactarO 

♦GIPSYCbaracterO 

♦getValuaO 

♦loStnngO 

♦getEnclosedTypeOjectO 



GIPSYFunction 

(from Idrg) 



* FUMCTIOM TYPE FUWCTIOWAL : int = 1 

J FUMCTION TYPE ST Int = 2 

j FUNCTIQH state immutable : int ^ 3 

.S FUMCTION STATE VOLATILE : int ^ 4 
^iFunctinnStata : int = FUNCTION_STATE_IMMUTABLE 
%.|FunctionType : int = FUNCTION_TYPE_FUNCTIONAL 



♦GIPSYFunclionO 

*GIPSYFunctionO 

♦galEnclosadTypaOjactO 

♦getValueO 

*teStringO 

*ga1Functior3tateQ 

*se1FunctiorStateO 

*ge1FunctlorTypeO 

*se1FunctiorTypeO 



^oFloatValue , Float 



♦GIPSYFIoatO 

♦GIPSYFIoatO 

♦toStringO 

♦GIPSYFIoatO 

♦getEnclosedTypeOjectO 

♦gatValuaO 



GIPSYStnng 



^oStrlngValue : String 



♦GIPSYStnngO 
♦GIPSYStringO 
♦gatValueO 
♦toStnngO 

♦getEnclosedTypeOjectO 



GIPSYBoolear 

Cfrom larg) 

^oBooleanValue : Boolean 

♦GIPSYBooleanO 

♦GIPSYBooleanO 

♦GIPSYBooleanO 

♦gatValueO 

♦toStnngO 

♦getEnclosedTypeOjectO 



♦GIPSYVoidO 



GIPSYType 



^strLe>i 
OIYPE. 
g TYPE 
a TYPE 
<? TYPE 
a TYPE 
a TYPE 
<> TYPE 
a TYPE 
a TYPE 
a TYPE 
^ TYPE 



;me : String - 
IHT int = D 



DOUBLE . int^ 1 
STRING : int^2 
BOOLEAN : int = 3 
CHARACTER int = 4 
ARRAY . int ^ 5 
OBdECT int ^ e 
VOID int = 7 
EMBED : lnt = 8 
IDENTIFIER Int ^ 9 



FLOAT: inl = 10 
int = - 1 

String = "<anonyrous>" 
a TYPE FUNCTION : int ^ 11 
■> TYPE OPERATOR int ^ 12 



♦getLexemeO 

♦toStrlrgO 

♦setLexemeO 

♦getTypoEnumeratlonO 

♦gGtEnclosedTypaOjoctO 

♦getlDO 

♦setlDQ 

♦getTypeO 



GIPSYObject 

^oObjectValue : Object 

♦GIPSYObjactO 

♦toStnngO 

♦OIPSYObjedO 

♦gatEnclosadTypaOjGctO 

♦getValueO 



#oBasGTypa 



GIPSYArray 



♦GIPSYArrayO 

♦OIPSYArrayO 

♦GIPSYArrayO 

♦toStnngO 

♦GIPSYArrayO 

♦lengthO 

♦getBaseTypeO 



^oEmbedValue : Object 



♦OIPSYEmbedO 
♦GIPSYEmbed(pstrURI String) 
♦GIPSVEmbad(poURI : URI) 
♦toStringO . String 
♦getEnclosedTypeOjectO ' Object 
♦getValueO : Object 



Figure 42: GIPSY Type System. 



the URL parameter and later is exploded into multiple types corresponding to STs 
and their CPs. GIPSYFunction and its descendant GIPSYOperator correspond to the 
function types for regular operators and user defined functions. A GIPSYFunction 
can either encapsulate an ordinary Lucid function (as in functional programming an 
which is immutable) or an ST function (e.g. a Java method), which may easily be 
volatile (i.e. with side effects). These four types are not directly exposed to a GIPSY 
programmer and at this point are managed internally. The rest of the type system 
is exposed to the GIPSY programmer in the preamble of a GIPSY program, i.e., the 
#funcdecl and #typedecl segments, which result in the embryo of the dictionary 
for linking, semantic analysis, and execution. Once ST compilers return, the type 
data structures (return and parameter types) declared in the preamble are matched 
against what was discovered by the compilers and if the match is successful, the link 
is made. 
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4.1.1.6 GICF Design 

The GICF is tlie first generalization framework of liybrid programming in the GIPSY. 
Implementation-wise, only Java is implemented as an imperative language with an 
external compiler. However, provision was made for C/C++, Perl, Fortran and 
Python with stub compilers. The class diagram describing GICF is shown in Fig- 
ure 43 On this diagram the interaction between a given imperative compiler and the 
SequentialThreadGenerator and CommunicationProcedureGenerator only shown 
for JavaCompiler to keep the clearer picture, but the same kind of association will 
have to be maintained for all imperative compilers as the IlmperativeCompiler in- 
terface mandates. The ElmperativeLanguages is a Java interface enumerating all 
available imperative language compilers. It is used by the GIPC to discover a given 
compiler for a language dynamically. As of this writing, the enumeration is main- 
tained by hand; however, it is planned to be generated in the near future with a 
command-line-driven script or a RIPE GUI automagically to facilitate addition of 
new languages. 

4.1.1.7 Intensional Programming Languages Compiler Framework 

As a consequence of GICF, a similar approach was applied to the intensional com- 
pilers in the form of IPLCF. See the corresponding class diagram in Figure |44j The 
IlntensionalCompiler was designed and implemented by all the intensional compil- 
ers we have. An enumeration EIntensionalLanguages of all supported intensional 
languages was created, so the GIPC can pick needed compiler at run-time as deter- 
mined by the Preprocessor. 

Translation for all intensional compilers is done through the generic Translator 
implemented by Aihua Wu in |Wu02] . The Translator has been integrated into 
the GIPC . intensional . GenericTranslator package and split and renamed as in 



Figure 45 Thus, every SIPL compiler refers to this translator to acquire a GIPL 
AST at the end via generic implementation of IntensionalCompiler. translate (). 
The Translator was refactored and augmented to understand GIPSY Types (see Sec- 
tion 4.1.1.5) and ImperativeNode for imperative languages. The TranslationParser 
and TranslationLexer collaborate to compile intensional language translation rules 
(e.g. IndexicalLucid.rul) files provided by each SIPL author. 
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o 



o 



llmperativeCompiler 

(from imperdlive) 



ElmperativeLanguages 

(from imperative) 



ImperativeCompilerException 

(from imperative) 



ImpefativeCompi/er 



CPPCompiler 

(from Cpp) 




CCompiler 

(from C) 



FottranCompiler 

(from Fortra n) 



PerlCompiler 

(from Perl) 



PythorCompiler 

(from Pytiion) 



#oSTQenerator 



FormatTag 

(from imperative) 



#oFormatTag 
< 



ImperativeNode 






JavaCompiler 

(f ro m Java) 









CommunicationProcedureGenerator 

(from CommuriicaiioriProcedureG-eneraior) 



JavaCommunicationProcedureGenerator 

(■from CommunicaiionPrcicedure(jen eraior) 



SeqiientiaiThreadGenerator 



JavaSequentialThreadGenerator 

(from Java) 



Sequent iaIThreadSourceGenerat or 

ij rom Se quentialThreadOe nerator] 



Figure 43: GICF Design. 



4.1.1.8 Sequential Thread and Communication Procedure Interfaces 

This section details Sequential Thread and Communication Procedure interfaces. 



The related class diagram is in Figure 46 The ICommunicationProcedure and 
ISequentualThread are the core interfaces. Both extend Serializable in order 
for us to be able to dump their concrete implementations to disk or distributed stor- 
age using Java's object serialization machinery. This is needed for the GIPSYProgram 
container to be saved to disk or for an ST to be able to reside in JavaSpaces |Mam05] 
implementation of the demand space |VP05j . The ISequentialThread also extends 
Runnable to be true thread when materialized, especially for the case of local execu- 
tion. The Runnable interface makes it possible for an implementing class to become a 
thread in multithreaded environment in Java. The ICommunicationProceduresEnum 
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o 

llntensional 
Compiler 

(from intensi...) 

^ranslateQ 



O 



EInlensionalLanguages 

(from intensional) 



IntensionalCompilerException 

(from intensional) 



^IntensionalCompilerEjiceptionQ 
^lntensionalCompilerE>iceptionO 
^IntensionalCompilerE^iceptionQ 



GIPLCompiler 

(from GIPL) 

^GIPLCompiIeK) 
♦GIPLCompiler() 
♦GIPLCompiIeK) 

*parse() 



IndexicalLucidCorrpiler 

(from IndexicalLucid) 



^IncJefticalLucidCompilerO 

^IndeificalLucidCompilerO 

^IndesdcalLucidCompilerO 

^initO 

^parseQ 



JLucid Compiler 

(from JLucid) 



^JLucidCompilerQ 

^JLucidCompilerQ 

^JLucidCompilerQ 

^initO 

^parseQ 



IntensionaSCompiSer 

(from inlensional) 



^IntensionalCompilerQ 
^IntensionalCompilerQ 
^IntensionalCompilerQ 
^compileO 

et Abst ra ct S y nt a xTre e Q 
^ranslateO 
^runQ 

^setSourceCodeStreamQ 
^oStringO 
^getLastExceptionO 
^compileQ 




(from SenericTranslator) 



GLUCompilGr 

(from tjLU) 



^GLUCompilerQ 
^GLUCompilerO 
^GLUCompilerO 

^parseQ 



LucxCompiler 

(from Luck') 



OnyxCompiler 

(from Oniix) 



-oJLucidCompiler 



TensorLucidCompiler 

(from TensorLucid) 



^■TensorLucidCompilerO 

*'TensorLucidCompiler() 

^■TensorLucidCompilerO 

♦initO 

^parseQ 



Objective LucidCo mpiler 

(from Dbj ectiveLucid) 



^Object iveLucidCompilerQ 
^Object iveLucidCompilerO 
^Object iveLucidCompilerQ 
^initO 
^parseQ 



^LucxCompilerQ 
^LucxCompilerQ 
^LucMCompilerO 
♦initO 
^parseQ 



*OnyxCompiler() 
^OnyxCompiler() 
*OnyxCompiler() 
JinitQ 
*patseO 



\ / / 
\\ / / 



Unimplemented 1^ 
stubs. 

Figure 44: IPLCF Design. 

is an enumeration of all known to the GIPSY communication procedure types. The 
NullCommunicationProcedure and RMICommunicationProcedure represent concrete 
implementations for local threaded processing as well as RMI. Therefore, the 
SequentialThreadGenerator is an abstract factory for all sequential threads that 
has to be overridden by a language-specific sequential thread generator, e.g. such as 
JavaSequentialThreadGenerator. Likewise, CommunicationProcedureGenerator 
is a factory for CPs. The WorkResult class represents the result of (computation) 
work done, which is also has to be Serializable. Upon various communication 
needs the CommunicationStats is returned by the ICommunicationProcedure API 
or the CommunicationException is thrown indicating an error. The Worker class 
represents a collection of STs and CPs being executed. 
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Translator 

(from (jenericTranslatoO 



Translation It em 

(from (jenericTranslatoO 



Cr 



TranslationParser 

(from GenerioTranslator^ 



(from (jenericTranslator^ 






TranslationLexer 

(from GenericTranslato ^ 









T 

(from OenericTranslator^ 

Figure 45: SIPL to GIPL Translator Integration. 
4.1.1.9 GIPC Design 



In Figure 48 there is a hierarchy that all imperative and intensional compilers should 
adhere to. The IlmperativeCompiler interface is something every imperative com- 
piler implements to ease up the job of GIPC. A similar interface has been invented 
for intensional languages - IlntensionalCompiler for consistency. 

A set of interfaces has been designed for all the present and future compilers to 
implement. There are three interfaces so far: 

1. ICompiler is a superinterface for all compiler interfaces. It is implemented by 



GIPC itself and by DFGAnalyzer, as shown in Figure 47 



2. IlntensionalCompiler is a subinterface of ICompiler designated to differenti- 
ate intensional compilers. It is implemented in part by the IntensionalCompiler 
abstract class that most (for now all) intensional compilers implement. 

3. IlmperativeCompiler is a counterpart of IlntensionalCompiler. Its purpose 
is similar to that of IlntensionalCompiler for imperative languages. 
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NullCommunicationProcedure 

(from CommunicationProcedureOenerator) 



RMICommunicationP rocedure 

(from CommunicationProcedureOenerator) 



Worker 

(from wrappers) 



^orkerO 

^demandQ 

^receiveQ 

^orkQ 

^stopWorkerO 

^runQ 



SequentialThreadSourceGenerator 

(from Sequent! alThireadOe nerator) 



^SequentialThreadSourceGeneratorQ 
^generateQ 



ISequentialThread 

(f ro n/i nte riac es) 



-aoCommuncationProceduresh 



CommunicationP rocedure 

(from C ommunioalionProoedureGenerator) 



-o 




^orkQ 
^g^tWorkPesiiitO 
setM ethod 



CommunicationStatus 

(from interfaces) 



CommunicationException 

(from interfa ces) 



o 



IC ommunic ationProceduresEnum 

(from interfaces) 

ONULL_CP : int = 
<>RMI_CP : int = 1 
#CORBA_CP : int = 2 

♦COM_CP : int = 3 
#JINI_RMI_CP : int = 4 
#JINI_JERI_CP : int = 5 



JCommunicationProcedure 

(from i nterfa ces) 



^iriitO 
^openQ 
^closeQ 
^sendQ 
^receiveQ 
^getPeturnTypeO 
^getParamTypeQ 
^getParamTypesO 
^setPeturnTypeQ 
^setParamTypeQ 
^setParamTypesO 
^getParamTypeQ 
^getParamTypeQ 
^getParaiTiListSizeO 




O 

Runnable 

(from lang) 



*runO 



WorkResult 

(from interfaces) 



CorrimunicationProcedureGenerator 

(fro m C m m u n i cati o n P ro ce d u re Oe n e rate r) 



^oCPImplementations : Hashtable = ne^v Hashtable Q 



^CommunicationProcedureGeneratorO 
^generateQ 

^getCommunicationProceduresO 



Sequ ential ThreadGene rat or 



^Sequent la IThreadGeneratorQ 
^generateQ 

^getSe quenti alThre adsO 



JavaSequentialThreadGenerator 

(f ro m J ava) 



^generateQ 

^JavaSequentialThreadGeneratorQ 
^JavaSequentialThreadGeneratorQ 



JavaCommunicationProcedureGenerator 

(fro m C m m u n i cati onProcedureOene rate r) 



Figure 46: Sequential Thread and Communication Procedure Class Diagram. 



The core difference between IlntensionalCompiler and IlmperativeCompiler 
versus the general ICompiler is that most (except for GIPL) of the intensional com- 
pilers have to perform SIPL-to-GIPL translation; hence, the translate () method, 
and all imperative compilers must produce communication procedures and sequen- 
tial threads as the result of their work; hence, generateSequentialThreadsO and 
generateCommunicationProcedures methods are provided. The abstract classes 
IntesionalCompiler and ImperativeCompiler provide the most common possible 
implementation for all intensional and imperative compilers respectively, so the un- 
derlying concrete compilers only have to override some parts specific to the language 
they are to compile. If extension of these classes is not possible for some reason 
(e.g. when writing external GIPSY plugins when a compiler class already inherits 
from some other class), they must implement their corresponding interface. Out of 
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Figure 47: All GIPC Compilers. 
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the concrete classes on the diagram the author of this thesis fully implemented GIPC, 
GIPLCompiler, IndexicalLucidCompiler, JLucidCompiler, ObjectiveLucidCompiler, 
and JavaCompiler. The DFGAnalyzer of Yimin Ding was made to implement ICompiler 
as it in fact compiles the "DFG code" out of GIPL or Indexical Lucid. 

The overall design and integration of the GIPC participants is illustrated in Fig- 
ure 



48 The GIPC class is the main compiler application that drives the compilation 



process, so in the general case in invokes the Preprocessor, intensional and impera- 
tive compilers required, the SemanticAnalyzer, Identif ierContextCodeGenerator, 
Translator, and the GEERGenerator linker. It also acts like a facade to other GIPSY 
modules. The major data structures, such as AbstractSyntaxTree, Dictionary, 
CodeSegment, FormatTag, ImperativeNode, and SimpleNode are created, accessed, 
or modified throughout the modules during the compilation process. Out of impera- 
tive languages only JavaCompiler is mentioned as it is the most advanced in this cat- 
egory. The JLucidCompiler's JLucidParser underneath invokes both JGIPLParser 



and JIndexicalLucidParser as JLucid Section 3J^ provides extensions to both of 
these languages. A number of association links have been removed from the diagram 
to maintain clarity as these links are intuitive or present in detail diagrams. 



4.1.1.10 GIPC Class as a Meta Processor 

The GIPC (a concrete class) acts here as so-called "meta processor" that drives the 
entire compilation process and invokes appropriate submodules in order to come up 
with a compiled version of a GIPSY program. This involves calling the Preprocessor, 
then feeding its output to whatever concrete compilers for the code segments of the 
GIPSY program, collecting the output of them (various ASTs, dictionaries), per- 
forming semantic analysis, and linking all the parts back together in a binary form. 
This portable binary version of the GIPSY program is to either be serialized as an 
executable file for later execution by the GEE or optionally to be fed directly to the 
GEE. 



4.1.1.11 Calling Sequence 



The sequence diagram in Figure [49] illustrates the entire compilation process and the 
data structures passed between the modules. This is the roundtrip description of the 
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Figure 48: Overall GIPC Design. 
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implementation efforts. The two followup diagrams detail the differences in the com- 
pilation process between the imperative and intensional languages. The general com- 
pilation process begins by reading the source GIPSY program and converting it into 
a meta token stream of types, declarations, and code segments by the Preprocessor. 
The Preprocessor takes that input and with its own parser produces a preprocessor 
AST and an embryo of a dictionary with the identifiers and types declared in the 
imperative code segments for further semantic linking. The latter is used to pro- 
duce imperative stubs for cross-segment type checks. The former contains primarily 
code segments written in various languages. The GIPC takes these code segments 
and creates appropriate compiler threads, one for each code segment. Then, each 
compiler tries to compile its own chunk and produces a portion of a main AST. Since 
we treat the IPL part as a main program, its AST is considered to be the main 
skeleton tree. The ASTs produced by the imperative compilers (which really contain 
a single ImperativeNode) are secondary and should be merged into the main when 
appropriate. Once all the compiler threads are successfully done, the GIPC collects 
all the ASTs and performs linking via the GEERGenerator. The combined AST is 
now a subject to the semantic analysis and the function elimination. Once semantic 
analysis is complete, the final post-linking is performed where all the pieces of the 
GIPSYProgram are combined together and its instance is serialized to disk. Option- 
ally, right after compilation the GEE may be invoked to start the execution of the just 
compiled program. 

There is no any preference made in GIPC on the number and the order of inten- 
sional and imperative compilers executed. This may result in several main intensional 
programs (if the source code contained more than one intensional code segment) or 
unused imperative nodes (an imperative segment is declared but the code from it is 
unused). For the former we maintain an array of ASTs in the GIPSYProgram, so that 
when the actual program is executed, the same number of the GEE Executor threads 
are started and all main ASTs are evaluated in parallel providing the result set of a 
computation instead of a single result. Detailed sequence diagrams of the intensional 



and imperative compilation processes are in Figure 50 and Figure 51 to illustrate the 
differences in compiling intensional and imperative code segments. 



93 




Figure 49: Sequence Diagram of GIPSY Program Compilation Process. 
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Figure 50: Sequence Diagram of Intensional Compilation Process. 
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Figure 51: Sequence Diagram of Imperative Compilation Process. 



96 



4.1.1.12 Compiling and Linking 

Multiple Intensional Parts In a GIPSY program we may possibly have multiple 
intensional parts. For example, if a GIPSY programmer gave a GIPL expression, an 
Indexical Lucid expression and a couple of Java procedures in the same source GIPSY 
program, what is the meaning of that setup would be? In this case, we can say that 
we evaluate two independent intensional expressions in parallel that happened to 
share the same imperative part. Thus, for such a GIPSY program there will be two 
instances of GEE running. The GEE is to extended to accept a forest of ASTs to be 
processed in parallel. 

Imperative Stubs When the Preprocessor completes its job, it has to create 
some stubs in the intensional parts of the program for the symbols declared outside 
of those parts (e.g. Java functions) so that the appropriate intensional compiler 
does not complain about undefined symbols when producing the AST because the 
intensional compilers are not aware of anything outside their work scope. Later on, 
the corresponding stub nodes in the AST are found and replaced with the real contents 
at the linking stage. 

NCP Generator as a Type Processor The NCP generator will act very much 
like a type processor and will have to look inside the imperative code segments ana- 
lyzed/compiled by the ST generator. This kind of type processing is needed to decide 
on communication procedures (CPs) to be generated for that ST. It issues warnings if 
the compiled version of the data structures to be sent is not portable. The role of the 
NCP generators in the GIPSY implementation is played by the imperative compilers, 
such as JavaCompiler. 

GEER Generator as a Linker The GEER Generator (see GEERGenerator in 



Figure 53) in the backend acts like a linker of all parts of a GIPSY program. It 
gathers all the resources from the compiler set, such as ASTs, ICs, CPs, STs, and the 
dictionary. Then, it replaces the stubs in the intensional part with the nodes from the 
imperative ASTs (STs accompanied with their respective CPs) forming a complete 
composite AST ready for consumption by the GEE. All this will be serialized as a 
GIPSYProgram class instance. The GEERGenerator is invoked two times - first prior 
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Figure 52: Semantic Analyzer. 

SemanticAnalyzer to assemble a complete AST, and then after semantic analysis 
and function elimination to set up the finalized dictionary and program name. 

4.1.1.13 Semantic Analyzer 



The semantic analyzer detailed design diagram is shown in Figure |52] Originally 
implemented by Aihua Wu, the class was renamed from Semantic |Wu02j to a more 
complete name of SemanticAnalyzer and placed under the GIPC package. Relevant 
changes include integration of storage .Dictionary (previously was java.util .Vector) 
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storage .Dictionaryltem (formerly Item_in_Dict |Wu02j ) . storage . Functionltem 
(formerly Fun_Item |Wu02j . serves for function description). The SemanticAnalyzer 



had to be taught to recognize new GIPSY types (see Section 4.1.1.5) with base 
GIPSYType class for object, embed, and array processing, ImperativeNode for se- 
quential threads and communication procedures, and a general AbstractSyntaxTree. 

4.1.1.14 Interfacing GIPC and GEE and Compiled GIPSY Program 

Now, let us formally define the notion of a stored compiled GIPSY program, as a 
GEER or the interface between the two major modules - GIPC and GEE. Until this 
point, the GEE accepted from GIPC as the input AST of an intensional part and a 
dictionary of symbols. This suggests having serialized the AST and the dictionary. 
With the invent of JLucid, communication procedures (CPs) and sequential threads 
(STs) became relevant and should belong to the GIPC-GEE interface. Thus, a com- 
piled GIPSY program may have several of CPs and STs serialized along. While STs 
and CPs are present within imperative AST nodes, references to them are recorded 
here for quicker access and decision making by the GEE. Then, as GEE produces 
demands (especially over RMI or Jini, |VP05j ) for each intensional identifier in the 
dictionary an Identifier Context (IC) class created |LGP03l ILu04] . This is needed 
because every such identifier represents a Lucid expression to be evaluated by the 
engine, and as such should also be part of the compiled GIPSY program. The cor- 



responding class diagram is in Figure 53 It includes the GIPSYProgram and all its 



associations with GIPC, GEE, GEERGenerator, and the storage classes. 

To summarize, the GIPC-GEE interface is the GIPSYProgram representing encap- 
sulation of the five parts: 

1. Linked AST(s) 

2. Dictionary 

3. A set of STs 

4. A set of CPs 

5. A set of ICs. 

On the diagram in Figure [38] GIPSYProgram defines and corresponds to the GEER. 
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Figure 53: Class diagram describing GIPSYProgram. 
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Figure 54: JLucid Design. 



4.1.2 JLucid 
4.1.2.1 Design 



The class diagram describing JLucid is shown in Figure 54 The implementation of 
JLucid parser-wise is heavily dependent on that of Indexical Lucid as the largest chunk 
of the IPL work is the same. JLucid adds a preprocessor JLucidPreprocessor class 
that is responsible for parsing initial source JLucid program and extract Java and 
Lucid parts. The JLucidParser class is the one that manipulates javacc- generated 
parsers amended to support embed () and arrays. The sequence diagram describing 



the details of the compilation sequence of JLucid is presented in Figure |55 

JLucid implements generation of Java sequential threads (STs) and their com- 
munication procedures (CPs); thus, necessitating JavaSequentialThreadGenerator 
and JavaCommunicationGenerator. For uniformity, portability, and testing reasons, 
we also decided to send the source code over, that can possibly be compiled on the 
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Figure 55: JLucid Compilation Sequence. 

remote machine. All this is done by the GICF-integrated JavaCompiler, see Sec- 
tion 11X231 

4.1.2.2 Grammar Generation 

As it was shown in Chapter |3j the JLucid syntax extension to GIPL and Indexical 
Lucid is minimal. The JavaCC grammars we use, are stored in the . jjt files for the 
original two dialects. If we decide to have very similar grammar files for JLucid to 
support JLucid extensions (arrays and embedO), then if the original grammar has a 
bug, the fix will have to be propagated to all the derived grammars, which will not 
scale from the maintenance point of view as there will be similar small modifications 
from Objective Lucid and other dialects. Thus, it was decided to only maintain the 
original grammars of GIPL and Indexical Lucid and generate the ones for the dialects 
with the minimal changes, so that each dialect only maintains the part that is relevant 
to its syntactic extension. 

For JLucid three bash shell scripts were created to process the original JavaCC 
grammars of GIPL and Indexical Lucid and generate appropriate extended versions 
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Figure 56: Java Compilation Sequence. 

for JLucid. These include j lucid. sh that generates JavaCC productions for ar- 
rays and embedO, JGlPL.sh that alters the original GIPL. jjt grammar to suit the 
needs of JLucid mostly in terms of class and package names and the new pro- 
ductions. Similarly, the JlndexicalLucid. sh script exists for processing of the 
IndexicalLucid. j jt file. The scripts are rather small and presented in the Ap- 
pendix |D} 

4.1.2.3 Free Java Functions and Java Compiler 

As defined in Chapter |3} by "free Java functions" we mean is that the corresponding 
Java STs don't have an enclosing Java class as far as JLucid source code concerned. 
However, the enclosing class must exist when compiling a Java program according 
to Java's syntax and semantics. Thus, implementation-wise we generate such a class 



internally that wraps all our sequential threads, as e.g. in Section 4.1.1.8 and we com- 



pile that class. This job of wrapping is delegated to the JavaCompiler, a member 



of the imperative compilers framework (see Section 4.1.1.1). The JLucidCompiler 



as shown in Figure 55 at some point invokes the JavaCompiler, and what the 



JavaCompiler does internally is illustrated in Figure 56 



Being an imperative compiler, the JavaCompiler is obliged to produce the Java 
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STs and CPs among other things. The core of this process is the wrap ( ) method where 
the actual "wrapping" our pseudo-free Java functions into an internal class occurs. 
The generated source code . j ava file is saved and is fed to the external j avac compiler 
as of this implementation. If there was no compilation errors, a corresponding . class 
or series of . class files (for the case of nested classes) is generated. The generated 
classes are reloaded back by the JavaCompiler and their members that are of interest 
to us retrieved via the Java Refiection Framework |Gre05] , thus we obtain an array of 
references to the ST methods and their parameters and assign them to our own data 
structures. After this process completes, the corresponding FormatTag describing 
the Java language and the compiler is created and all information is embedded into 
the ImperativeNode, which represents a single and the only node in the imperative 
AbstractSyntaxTree. Later on, this imperative node or its pieces will replace a 
corresponding stub in the main intensional AST. 



4.1.2.4 Arrays 

Implementation of arrays in JLucid coincides closely with the implementation of ob- 



jects in Objective Lucid in Section 4.1.3 As a part of the GIPSY Type System (see 
Section 4.1.1.5), we employ the GIPSYArray (see Figure 42) type to hold the array 
base type and its members and an overall value. As proposed further, we treat ar- 
rays internally as objects (and objects as arrays), so GIPSYArray is an extension of 
GIPSYObject that has a base type asserting the data type of the all the elements in 
the arrays (as our arrays a homogenous collection of elements). Thus, when a syntac- 
tic array token is parsed, a corresponding instance of GIPSYArray is created to hold 
the type and value information for later processing. The SemanticAnalyzer and the 
Executor are made to understand the array type and apply similar type checking or 
execution rules to a collection of values instead of a single value. 

It might look like this approach will clash with the use of arrays in Java, i.e., 
when a developer wishes to use Java arrays (or if a library already implements some 
functionality via Java arrays). This should not be a problem (though will require 
a more thorough investigation in the future work), when we perform type matching 



by the base element type, as described in Section 4.1.1.4 The JavaCompiler is 
responsible for the appropriate conversion of the native-to-GIPSY type conversions, 
by supplying a TypeMap such that it can also be used by the GEE at run-time. Similar 
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comments can be said of the native array types that might exist in other imperative 
languages that we would be hoping to support. 



4.1.2.5 Implementing embedO 

To implement embed () we define a type GIPSYEmbed to fetch the file pointed by the 
URL and hold it in there. In JLucid, a .Java or .class file (later also a .jar file) 
is loaded from either local or remote location pointed by the URL as follows: if it 
is a .Java file, it's fetched and compiled similarly to the generated class, but the 
name is static and known; with the . class file we skip the compilation process, but 
extraction of the sequential threads is the same; for the .jar its examined with the 
JarlnputStream and JarEntry Java classes to extract the class information. 



4.1.2.6 Abstract Syntax Tree and the Dictionary 

When running the JLucid compiler in stand-alone mode, all the preprocessing and 
re-assembling the intensional and imperative pieces into the combined main AST 
happens in here, not in the GIPC, so the JLucid compiler returns a complete linked 
AST with all imperative nodes linked in place and a proper dictionary of identifiers, 
both intensional and imperative. JLucid compiler, however, reused the Preprocessor 
and other parts of the new framework internally instead of re-inventing the wheel. 

The JLucidPreprocessor uses the general Preprocessor class to do the job of 
chunkanizing the code segments and preparing initial imperative stubs. This neces- 
sitated adding the #funcdecl segment in the JLucid programs that previously did 
not have one in Chapter [3} to simplify preprocessing and generation of the dictio- 
nary. The JLucidPreprocessor is set to reject any other code segments than #JAVA, 
#JLUCID, or #funcdecl. 

If the JLucidCompiler invoked from the GIPC as a part of general compilation 



process (see Figure 49), the #JAVA segment will no longer be really processed inter- 
nally, and instead, GIPC will call JavaCompiler externally to the JLucidCompiler, 
so essentially the JLucidCompiler will be responsible only for the Lucid part (with 
arrays and embed ()). 
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Figure 57: Objective Lucid Design. 



4.1.3 Objective Lucid 

This section addresses problems that arise when implementing Objective Lucid. These 
include internal implementation to support the dot-notation, extension to semantic 
analysis to be able to manipulate object data types (very likely user-defined), and 
making it all work in the GICF and General Eduction Engine (GEE) of the GIPSY 
by correctly forming the abstract syntax tree (AST) that includes object data types. 
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Figure 58: Objective Lucid Compilation Sequence. 



4.1.3.1 Design 



The class diagram describing Objective Lucid is in Figure 57 Since the JLucid 
compiler already does most of the legwork, Objective Lucid simply extends it to add 
the dot-notation and some extra post-processing when unrolling the objects. The 



corresponding compilation sequence is shown in Figure 58 



4.1.3.2 Grammar Generation 



Like with JLucid, the grammar files are generated for Objective Lucid using bash shell 
scripts, ObjectiveGIPL.sh and ObjectivelndexicalLucid. sh. These scripts work 



with the grammars produced by the JLucid scripts (see Section 4.L2.2) by simply 



extending them with the dot-notation production and fixing up names of classes and 



packages. These scripts are presented in the Appendix D 



4.1.3.3 Object Instantiation 

Normally, when a Lucid program refers to a Java object, it has to instantiate it first 
by either calling a pseudo-free Java function that returns an object instance or to call 
the constructor directly. This instantiation has to be explicit at the beginning of the 
program to avoid Java's NullPointerException at run-time. Internally, the object 
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instance is created using Java Reflection |Gre05j by first loading and then initializ- 
ing the needed class with Class . forNameC'ClassXB") .newlnstanceO. Referencing 
static members do not require a class instance, and can be accessed using the class 
name, in this case we just keep the Class .forNameC'ClassXB"). We also keep the 
needed references to the object itself and its members in the GlPSYObject type of 
the GIPSY Type System. 



4.1.3.4 The Dot-Notation 

Implementing the dot-notation extension of JLucid is the easiest task of the three. 
In fact, the E. id productions are just a syntactic sugar that can be wrapped around 
already existing mechanisms of JLucid to include Java functions as mentioned in 



Section 3.2.1.1 The compiler simply generates a set of pseudo-free Java functions for 
every object member referenced from the intensional program. These will be easy to 
place into the AST just the way JLucid does it. In other words, this is achieved by 
automatic generation of implicit accessor Java functions that had to be explicit in 
JLucid. 



4.1.3.5 Abstract Syntax Tree and the Dictionary 

The GIPC (General Intensional Programming Compiler) generates abstract syntax 
trees (AST) of all compiled GIPSY program parts, and constructs the GEER (General 
Eduction Engine Resources), which is a data dictionary storing all program identi- 
fiers, encapsulated with all ASTs generated at compile time. Simply put, the GEER 
encapsulates all the meaning of a GIPSY program, and all necessary resources to en- 
able the GEE to execute the programs correctly. The AST and the dictionary contain 
the generated accessor identifiers that are processed by the JLucid mechanisms, as 
described previously. This is possible because Java's built-in class Class can provide 
us with all the meta-information about its members through enumeration that we can 
place in the AST and the dictionary. Little changes from the way JLucid processes 
that except that the object members are put into the dictionary and acted upon as 
an array of homogeneous types as described in the follow up section. 

The ObjectiveLucidPreprocessor also makes use of the general Preprocessor, 
but unlike JLucidPreprocessor, it also accepts the #typedecl segment as with ob- 
jects come user-defined types, so these have to be hsted if used by the Lucid part. 
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4.1.3.6 Objects as Arrays and Arrays as Objects 

Implementation-wise, we propose to treat arrays of JLucid as a special case of objects 
and, the other way around, the objects be a generalization of arrays. An array can 
be broken into its elements where every element is evaluated as an expression under 
the same context. Thus, evaluating: 

A[4] a [d:4] 
where 

dimension d; 

A[#.d] = 42 * #.d fby.d (#.d - 1); 

end; 

is equivalent to evaluating four Indexical Lucid expressions (possibly in parallel). 
Under this point of view objects can be viewed as arrays where every atomic member 
is evaluated as if it were an array element. Basically, we denormalize an object into 
primitives and evaluate them. If an object encapsulates other objects, then these are 
in turn denormalized and put into the definition environment (dictionary). In other 
words, if you have an array of four elements a [4] , the elements are evaluated as four 
independent expressions. Likewise, an object that has four data members, each of 
them is evaluated as an expression under the same context. 

Essentially, an array is a collection of atomic elements of the same type. When 
evaluating say an array of four elements a [4] at some context [d:4] , we are, in fact, 
evaluating four ordinary Lucid expressions (possibly in parallel) in the same context. 
Likewise, an object is a collection of atomic elements of (possibly) different types. In 
case an object encapsulates another object, that other object can in turn be split into 
atoms, and so on. All atoms of an object evaluate as independent Lucid expressions, 
just like array elements. 
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Thus, from Objective Lucid's point of view, the following are equivalent: 



(a) int a [4] ; 



(b) class foo 

{ 

int al; 
int a2; 
int a3; 
int a4; 

} 



So, internally, we represent (a) in the definition environment as: 



a_4 // scope identifier 
a_4 . al 
a_4 . a2 
a_4 . a3 
a_4 . a4 



Under the scope of array a_4 (a generated id) there are four members, and a_4.a* 
comprise a denormalized identifier, also generated. And (b) will become: 



foo // scope identifier 

f oo.al 

foo.a2 

foo . a3 

f oo.a4 



where foo. a* are generated variable identifiers in the definition environment. En- 
capsulation will be handled in the following way: 



class bar 
{ 

int bl; 
int b2; 



foo oFoo = new foo(); 



} 



bar 




bar .bl 




bar.b2 




bar. foo 




bar .foo 


al 


bar . foo 


a2 


bar. foo 


aS 


bar. foo 


a4 



To paraphrase and explain in another example, if we have three separate Lucid ex- 
pressions: 



// float 
a [d:2] 
where 

dimension d; 

a = 2.5 

fby.d (a + 1); 

end; 



// integer 
b a [d:2] 
where 

dimension d; 

b = 1 

fby.d (b + 1); 

end; 



// ASCII Char 
c a [d:2] 
where 

dimension d; 

c = 'a' 

fby.d (c + 1); 

end; 
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Now if we group a, b, and c as a class: 



class foo 

{ 

float a = 2.5; 
int b = 1; 

char c = ' a ' ; 
public fooO {} 

} 



So when we write: 



f a [d:2] 
where 

dimension d; 

f = fooO fby.d (f + 1); 

end; 



WO mean there start three subexpression evahiations: 



f .a [d:2] 
where 

dimension d; 

f .a = f oo() . a 
fby.d (f.a + 1); 

end; 



f.b [d:2] 
where 

dimension d; 

f .b = fooO .b 
fby.d (f.b + 1); 

end; 



f.c [d:2] 
where 

dimension d; 

f.c = f oo () . c 
fby.d (f.c + 1); 

end; 



We say these are equivalent where the f in all expressions refers to the same object's 
instance (i.e. there are not three objects constructed, only one). Similarly (nearly 
identically) we implement arrays: 



a[3] a [d:2] 
where 

dimension d; 

a = [1, 2, 3] fby.d (a + 1) ; 

end; 



The above means: 



array a 
{ 



int al = 1; 
int a2 = 2; 
int aS = 3; 
int length = 3; 
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al @ [d:2] 
where 

dimension d; 

al = 1 fby.d (al + 1) ; 
end; 



a2 a [d:2] 
where 

dimension d; 

a2 = 1 fby.d (a2 + 1) ; 
end; 



a3 @ [d:2] 
where 

dimension d; 

a3 = 1 fby.d (a3 + 1) ; 
end; 



The three subexpressions run in parallel, but refer back to the same array. Should 
there be a need in one of the three subexpressions to use an array value produced by 
another subexpression, they generate a demand for that value. 



4.2 External Design 

The external design encompasses user interface design as well as external software 
interfaces. In this work, a web interface to the GIPSY as well as command-line 
interfaces are presented as a part of UI followed by the API of the two external 
libraries used, JavaCC and MARF. 



4.2.1 User Interface 

4.2.1.1 WebEditor - A Web Front-End to the GIPSY 

The user interface designed for the GIPSY in the scope of this thesis includes a Servlet- 
driven web interface to the GIPSY daemon server running on our development server 
for trying out GIPSY programs online. The web interface in a form of a web page 
allows a connected user to enter, compile, run, and trace GIPSY programs. Users are 
able to submit their own GIPSY programs (in any supported Lucid dialect) or choose 
and modify from existing programs from the GIPSY CVS repository (see |RG05a] ) 
and then launch the computation. The GIPSY servlet front-end generates demands 
through RIPE and returns back results along with an execution trace to a web form. 



A screenshot of this interface is illustrated in Figure 59 
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Go Bookmark; T00I5 Help | 



:5, Concordia, «:8a8aiWebEditori5er¥letiWebEditor 



73 ® » [E~ 



GIPSY WebEditor Portal 



Prototype $Revision: 1.25$ 

$Id: WebEditor.]ava,v 1.25 2005/09/14 03:41:07 mokhov Exp < 



Computer Science and 
Software Engineering 



Intensional Dialect: 



I GIPSY 



Test Programs^ 



[tests/gipsy/car-oop.ipl 



Load 



Controls: 

Parse | 
Run I 

Regressior Tests | 



JUnit Tests 



Stop I 
Print I 

R debug mode 



StaitGEESeivices | 
r Ali 

F Tlireaded (locai) 

r RMi 

r Jini 

r DCOM+ 

r CORBA 

r GIPSY Sockets 
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Method: POST 
Request URI: 

/WebEditor/serviet/WebEditor 

Protocol: HTTP/1,1 

Pathlnfo: null 

Remote Address: 

132,205.44,132 

Remote Host: 

a I f re d o . cs . conco rd i a . ca 



Program Text: 



SIHDEXICALLUCID 
fib g.t 5 

dimension t; 
fib = fby.t g: 

g = 1 fby.t [fib ■ 



Output: 



rpreted dialect: GIPSY 



fib = □ fby.t g; 
g = 1 fby.t [fib ■ 



1^ 



Figure 59: GIPSY WebEditor Interface. 
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4.2.1.2 GIPSY Command-Line Interface 

Synopsis: 

gipsy [ OPTIONS ] 
gipsy — help I -h 

This is an all-entry point for all of GIPSY that bundles all the modules. It 
generally passes all the options to RIPE for further dispatching. When the server part 



see Section 7.10) is complete, this will be a GIPSY daemon server. The command 



line interface includes the following options: 

• — help or -h displays application's usage information. 

• — compile-only tells to compile a GIPSY program only and return the result 
of the compilation (error or success messages) and the compiled program itself. 
This will not invoke the GEE for execution after compilation. The option is 
primarily for quick tests in development setups. 

• — debug tells to run in the debug/verbose mode. 

It is possible to run the GIPSY by either invoking the GIPSY . class directly, by run- 



ning a corresponding gipsy, jar (see Appendix C.2) file, or using a provided wrapper 
script gipsy. The latter is the simplest one to use as it includes all the necessary op- 
tions for the JVM and searches for the executable . j ar in several common places. A 
good idea is to put gipsy somewhere under one's PATH. (A similar approach applies 
to the other tools mentioned in the follow up sections, such as ripe, gipc, gee, and 
regression. The tools exist for both Unix and Windows in the form of shell scripts 
and batch files.) 

Example uses of the GIPSY application include: 

• gipsy or gipsy — help 

• gipsy — compile-only 

• gipsy — compile-only — debug 

Where — debug can be combined with any of these, otherwise the options are 
exclusive. 
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4.2.1.3 RIPE Command-Line Interface 

Synopsis: 

ripe [ OPTIONS ] 
ripe — help I -h 

Ttie RIPE command-line interface right now acts mostly to activate various own 
submodules (e.g. textual or DFG editors) or dispatch requests from users to the 
other main modules, such as GIPC and GEE. The command-line interface includes the 
following options: 

• — help or -h displays application's usage information. 

• — gipc='<GIPC OPTIONS>' tells RIPE to invoke GIPC with a set of GIPC op- 



tions (see Section 4.2.1.4). 



— gee='<GEE OPTIONS>' tells RIPE to invoke GEE with a set of GEE options 



(see Section 4.2.1.5). 



— regression= ' <REGRESSIOM OPTIONS> ' tells RIPE to invoke Regression test- 



ing with a set of its options (see Section 4.2.1.6). 



• — dfg='<DFG EDITOR OPTIONS>' tells RIPE to start the DFG editor with its 
options. Currently, the DFGEditor Java class is a stub, and instead, the DFG 
Editor of Yimin Ding |Din04j is started via a separate program, lefty. It is 
planned the DFGEditor class would be a wrapper for the program in the future. 
Therefore, all DFG editor options are ignored for now, but a provision is made 
for the future. 

• — txt='<TEXTUAL EDITOR OPTIONS>' tells RIPE to start the textual editor 
with its options. Note, at the time of this writing TextualEditor is just a 
stub, and as such does not have any options, but a provision is made when it 
does. 

• — debug tells to run in the debug/verbose mode. 
Example uses of the RIPE application include: 
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• ripe or ripe — help 

• ripe — compile-only 

• ripe — compile-only — debug 

4.2.1.4 GIPC Command-Line Interface 

Synopsis: 

gipc [ OPTIONS ] [ FlLENAMEl . ipl [ FILENAME2 . ipl ] ... ] 
gipc — help I -h 

The command line interface for GIPC inherited some options from Lucid |Ren02] 
and includes the following options: 

• — help or -h displays application's usage information. 

• [FlLENAMEl . ipl [F1LENAME2 . ipl] . . .] tells GIPC to compile a GIPSY pro- 
gram as indicated by the FILENAME. It is possible to have more the one input 
file for compilation. If this is the case, the same number of instances of GIPC 
threads will be initially spawned to compile those programs. Notice, however, 
this does not mean all the files (in case of multiple . ipl files) comprise one 
program and then linked together afterwards as in typical C or C++ compilers. 
Instead, each . ipl file is treated as a stand-alone independent GIPSY program. 

• — stdin tells GIPC to interpret the standard input as a source GIPSY program. 
This is the default if no FILENAME is supplied. 

• — gipl or -G (came from Lucid |Ren02j for backwards compatibility) tells GIPC 
to interpret the source program unconditionally as a GIPL program (by default 
no assumption is made and GIPC attempts to treat the incoming source code 
as a general GIPSY program). It is primarily used to quickly test the GIPL 
compiler only, without extra overhead or preprocessing. It is also used by the 
Regression application for that same reason. 

• — indexical or -S (came from Lucid |Ren02j ) tells GIPC to interpret the source 
program unconditionally as an Indexical Lucid program. 
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• — j lucid tells GIPC to interpret the source program unconditionally as a JLucid 
program. 

• — objective tells GIPC to interpret the source program unconditionally as an 
Objective Lucid program. 

• — translate or -T (came from Lucid |Ren02j ) enables SIPL-to-GIPL transla- 
tion. This option is implied by default (as opposed to be optional in Lucid). 
It tells the GIPC to interpret the input program unconditionally as a non-GlPL 
program that requires operator and function translation. The option has no ef- 
fect with — gipl as GIPL is the only intensional language that does not require 
any further translation. 

• — disable-translate turns off automatic translation (in case the user knows 
that an incoming non-GIPL program has nothing to translate, which is rarely 
the case; otherwise, the GIPC will bail out with an error). 

• — warnings-as-errors tells to treat compilation warnings as errors and stop 
compilation after displaying them. 

• — gee tells GIPC to run the compiled program immediately after compilation (if 
successful) by feeding it directly to the GEE. The default is that the compiled 
GIPSY program is saved into a file where the original name is suffixed with the 
. gipsy extension. 

• — df g tells GIPC to perform DFG code generation as a part of the compilation 
process. 

• — debug to run in a debug/verbose mode. 
Example uses of the GIPC application include: 

• gipc or gipc — help or gipc -h 

• gipc life.ipl 

• gipc — disable-translate — gee — debug life.ipl 

• gipc — gipl — debug gipl.ipl 

• gipc — j lucid — stdin 
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4.2.1.5 GEE Command-Line Interface 

Synopsis: 

gee [ OPTIONS ] [ FILENAMEl .gipsy [ FILENAME2 . gipsy ] ... ] 
gee — help I -h 

The command line interface includes the following options: 

• — help or -h displays application's usage information. 

• [FILENAMEl. gipsy [FILENAME2 . gipsy] ...] tells GEE to run a stored version 
of a compiled GIPSY program as indicated by the FILENAME. It is possible to 
have more than one input file for execution. If this is the case, the same number 
of instances of GEE threads will be initially spawned to run those programs. 
The programs will run concurrently, but there should not be any interference or 
communication in their execution except they may share the output resource. 

• — stdin tells GEE to interpret the standard input as a compiled GIPSY program. 
This is the default if no FILENAME is supplied. 

• — all tells GEE to start all implemented services/servers locally (threaded, RMI, 
Jini, DCOM+, and CORBA). 

• — threaded tells GEE to start the threaded server only. 

• — rmi tells GEE to start the RMI service. 

• — j ini tells GEE to start the Jini service. 

• — dcom tells GEE to start the DCOM+ service. 

• — corba tells GEE to start the CORBA service. 

• — debug tells GEE to run in the debug/verbose mode. 

Example uses of the GEE application include: 

• gee or gee — help or gee -h 

• gee life. gipsy 
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• gee — disable-translate — threaded — debug life. gipsy 

• gee — all — debug gipl. gipsy 

• gipc — rmi — jini indexical .gipsy 

4.2.1.6 Regression Testing Application Command-Line Interface 

Synopsis: 

regression [ OPTIONS ] 
regression — help I -h 



The Regression application and its test suite are presented in detail in Section 5.1 
The application, based on options, invokes either GIPC or GEE or both directly feeding 
a pre-selected list of test source programs. The command line interface includes the 
following options: 

— help or -h displays application's usage information. 
— sequential tells to run sequential tests (default). 
— parallel tells to run parallel tests. 
— gipl tells to test pure GIPL programs only. 

— indexical tells to test pure GIPL and Indexical programs with the Indexical 
Lucid compiler. 

— gipsy tells to test general-style GIPSY programs with code segments. 

— gee if specified, tells to run the GEE after compilation (default). 

— all tells to do all of the above tests in one run (default). 

— directory tells to pick source test files from a specified directory instead of 
pre-set directories from the GIPSY source tree 

— debug tells to run in the debug/verbose mode. 
Example uses of the Regression application include: 
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• regression or regression — help or regression -h 

• regression — gipl 

• regression — parallel — indexical 

• regression — all — debug 

• regression — directory=/some/gipsy/misc/tests — all — debug 



4.2.2 External Software Interfaces 
4.2.2.1 JavaCC API 

JavaCC-generated code contains a number of common classes and interfaces, regard- 
less of the language a parser is generated for. These have to do with AST nodes, 
tokens, token types, character streams, and alike. The most often used class out of 
this bundle is SimpleNode, which is a concrete node in the AST. These classes have 
to be periodically refreshed by compiling the source grammar when a newer version 
of javacc comes out. 

The below are JavaCC API/modules |VC05j used by the GIPSY and their de- 



scription. The corresponding class diagram is in Figure 60 



Node is the common interface for all occurrences of SimpleNode to implement 
(see below). 

The SimpleNode class represents a concrete node in every AST in the GIPC. 
Once generated, this class is usually customized according to the needs of the 
given parser/compiler. All concrete instances, however, implement the same 
Node interface above. At the time of this writing, there are three SimpleNode oc- 
currences in the GIPSY source tree: the common one in gipsy . GIPC . intensional 
for all the SIPLs and GIPL, as per original implementation presented in jRen02] . 
It is a basis for a GIPL AST aside from the related parsers known to the 
SemanticAnalyzer and GEE's Executor. This implementation is wrapped- 
around by AbstractSyntaxTree that the rest of the modules know. Then, a 
customized subclass of it is in gipsy. GIPC. DFG.DFGAnalyzer of Yimin Ding 
|Din04j . It was made a subclass because a large portion of the code is iden- 
tical. Finally, the last one is in gipsy. GIPC. Preprocessing used by the 
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;ure 60: JavaCC- and JJTree-generated Modules Used by Several GIPC Modules. 
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Preprocessor. This occurrence of SimpleNode was kept as-is due to the sig- 
nificant difi^erences and purpose with the former two. 



The ImperativeNode is another implementation of the Node interface created 
manually for all the imperative language compilers. The ImperativeNode rep- 
resents an AST of a single node encapsulating STs, CPs, some meta information 
that came from a given imperative compiler. The reason for this is to main- 
tain a global AST for a GIPSY program where all nodes implement the same 
interface. 

SimpleCharStream is a common javacc utility that treats incoming source code 
stream as a set of ASCII characters without extra UNICODE processing. 

ParseException is a common generated type of exception to indicate a parse 
error. It was made manually to subclass GIPCException from the GIPSY Ex- 



ceptions Framework (see Section 4.2.3.2) for uniform exception handling. 



• TokenMgrError a subclass of j ava . lang . Error primarily to signal lexical errors 
in the incoming source code or token processing in general by a given parser 
(e.g. by invoking a static parser twice). 

4.2.2.2 MARF Library API 



MARF (see Section 2.6.3 ) has a variety of useful utility and storage-related modules that 
conveniently found their place in GIPSY. Most of these come from the marf .util 
package as well as marf . Storage]^ The below are MARF API/modules used by 
GIPSY and their description: 

• marf .util .FreeVector is an extension of Java. util .Vector that allows the- 
oretically vectors of infinite length, so it is possible to set or get an element of 
the vector beyond its current physical bounds. Getting an element beyond the 
boundaries returns null, as if the object at that index was never set. Setting an 
element beyond bounds automatically grows the vector to that element. In the 
GIPSY, marf .util .FreeVector is used as a base for our Dictionary as shown 



in Figure 62 Figure 63 shows all the modules that are now using Dictionary 



^Later some natural language processing (NLP) modules in marf . nip of MARF might also get 
used in the GIPSY as a part of another research project. 
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Figure 61: MARF Utility Classes used by the GIPSY, 
instead of j ava . util . Vector. The corresponding class diagram of the MARF's 



util API is shown in Figure 61 



marf .util . OptionProcessor module is extensively used by the command-line 



user interfaces (see Section 4.2.1) of GIPSY, GIPC, GEE, and Regression. A con- 
venient way of managing command-line options in a hash table and validating 
them. 

marf .util .BaseThread class encapsulates some useful functionality used in 
threaded versions of GEE and GIPC, which Java's java. lang. Thread does not 
provide: 

— maintaining unique thread ID (TID) among multiple threads and reporting 
it (for tracing, debugging, and RIPE). A note is added here that Java 1.5.* 
now also provides a notion of a TID, but marf .util .BaseThread was 
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Dictionaryltem 
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♦getRanl<0 

♦setRanl<0 
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Figure 62: Dictionary and Dictionaryltem API 



written prior to that and GIPSY remains Java 1.4-compliant still. Plus, 
MARF's way of handling this is more flexible. 

— adapted human-readable trace information via toStringO 

— access to the Runnable target that was specifled upon creation. 

— integration with marf .util.ExpandedThreadGroup, see below. 



• marf .util .ExpandedThreadGroup allows to start, stop, or other group opera- 
tions that Java's j ava . lang . ThreadGroup doesn't provide. ExpEindedThreadGroup 
is, for example, used in GIPC to create a group of compiler threads (in GIPSY 
every compiler is a thread), one for each language chunk, that will run concur- 
rently. Additionally, a group of GEE, or rather. Executor threads would run in 
the case of a forest of ASTs. 



• marf .util .Arrays groups more array-related functionality together than the 
Java, util .Arrays class does, for example copying (homo- and heterogeneous 
types) and converting to Java. util .Vector, and provides some extras. 
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Figure 63: Dictionary Usage within the GIPSY 



• marf . Storage . StorageManager provides basic implementation of the (possibly 
compressed) object serialization, and in our case the GIPC and GEE are storage 
manager with respect to a compiled GIPSY program. 

• marf . util . Logger is primarily used by the Regression application to log stan- 
dard output before calling GIPC or GEE to a file, for future comparison with an 
expected output. 

• marf .util .Debug is used in many places for debugging convenience allowing 
to issue debug messages only if the debug mode is globally on, which is also 
maintained within the class. 
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4.2.2.3 Servlets API 

The Java Servlets technology from Sun jMicOSa] was used to implement the WebEditor 
interface outlined earlier. While the actual API specification of servlets is rather vast, 
the key used components used here are listed: 

• The HttpServlet class is the base for all servlets, including WebEditor. 

• The doGetO must be overridden to respond to the GET HTTP requests. 

• The doPostO must be overridden to respond to the POST HTTP requests. In 
our implementation, doPostO is a simply a wrapper around doGetO, so both 
GET and POST requests are handled identically. 



4.2.3 Architectural Design and Unit Integration 

Unit integration according to the initial design decisions of the GIPSY system and 
setting up package hierarchy played an important role in the success of this work. 



A proposed directory structure (see Appendix C.l) and a corresponding breakdown 



of the Java packages (see Appendix C.l) hierarchy are important to the success of 
GIPSY, especially for public use. The author of this work inherited the previous 
GIPSY iteration without any structure or packaging and proposed and restructured 
the system to what it is now. 



4.2.3.1 GIPSY 

When integrating several components of a large system and redesigning some of their 



API, the overall system design has to be considered. In Figure [64] is a high-level 
view of the main GIPSY modules. These modules can be run as stand-alone Java 
applications or start each other. 

• The GIPSY class on the diagram represents a stand-alone server for a client- 
server type of application, which is capable of spawning GIPC and GEE upon 
client's request. The prime goal of it is testing of intensional programs that users 
can submit online and get the result in case they don't have the development 
environment set up from where they are working. 
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These classes 
represent the main 
applications as 
well as tacades to 
the submodules of 
each subsytem. 
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Figure 64: GIPSY Main Modules. 



• The GIPC class when run as a stand-alone application invokes all the intensional 
and imperative compilers required and produces a compiled version of a sub- 
mitted GIPSY program. Optionally, if requested, GIPC can pass the compiled 
program on to GEE for execution. The GIPC along with GEE subsumes what was 
previously known as Lucid and Facet defined by Chun Lei Ren in |Ren02j . 

• The GEE when run as a stand-alone application, begins demand-driven execution 
of a GIPSY program that was either compiled and stored or compiled and passed 
from GIPC. 

• The Regression class is the main driver for the Regression Testing Suite of 
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Figure 65: GIPSY Exceptions Framework. 
GIPSY, that also calls these modules for regression and unit testing. 
4.2.3.2 GIPSY Exceptions Framework 

The class diagram describing the GIPSY Exceptions Framework is in Figure [65| The 
main exception type is GIPSYException that provides some machinery encapsulating 
other exceptions. Every major module, like GIPC, GEE, or RIPE in GIPSY de- 
fines its own sublcass of GIPSYException. By doing this, the applications using the 
modules can differentiate the exception types and handle them appropriately. The 
NotlmplementedException is an easy way to use to indicate some unimplemented 
but important stubs, if called. It is a subclass of Runt imeExcept ion because it can 
happen virtually everywhere and run-time exceptions do not need to be declared to 
be thrown or caught. The GIPCException, GEEException, and RIPEException rep- 
resent base exception objects for the corresponding modules; the rest are primarily 
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subclasses of these. 



4.2.3.3 GEE Design 



The general overview of GEE is in Figure |66j The several modules under the 
gipsy. GEE package carry out a complex GIPSY program execution task. The GEE 
is the facade and the main starting point for all of GEE. GEE may act as either an 
application on its own or be invoked by the GIPC. For the stand-alone execution a 
user has to supply a filename of a valid compiled GIPSYProgram. This program is 
loaded and GEE starts the Executor thread to actually execute it. Before Executor 
begins the GEE may optionally start the available demand propagation services, such 
as local (just threads), RMI, Jini-based and the like. The Executor while executing 
the program generates demands for the identifiers listed in the program and then per- 
forms the final calculation based on the results received. The Executor was formerly 
known as XLucidlnterpreter and the Java version of which was implemented by Bo 
Lu in |Lu04j and reworked to handle sequential threads, arrays, objects, and other 
than integer and fioat data types. 



Demand Dispatcher In Figure 67 is a high-level overview of the DemandGenerator 

and related classes. Most of the demand propagation in Jini and JavaSpaces is im- 
plemented by Emil Vassev in |VP05] . The integration part included making sure the 
IDemandList interface is consistently used by the DemandGenerator along with the 
DemandDispatcherAgent to be compliant to the rest of the GEE. The IDemandList 
interface was originally designed by Bo Lu in |Lu04j and redesigned by the author 
of this thesis to be implemented by the RMI and threaded versions of GEE and was 
formerly known as DemandList. Next, the temporary class WorkTask was made to im- 
plement the ISequentialThread interface according to the overall GIPSY design for 
sequential threads. This class is marked as deprecated (and later on will be removed) 
as every sequential thread class is generated by the SequentialThreadGenerator and 
is different from one GIPSY program to another. Finally, the LUSException (service 
look up exception) and DemandDispatcherException were made to be a part of the 



GIPSY Exceptions Framework Section 4.2.3.2 by inheriting from the GEEException 



For further implementation details of the DemandDispatcher please refer to Emil's 
work |VP05j . 
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Figure 66: GEE Design. 
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Figure 67: The Demand Dispatcher Integrated and Implemented based on Jini. 
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Figure 68: Integration of the Intensional Value Warehouse and Garbage Collection. 



Intensional Value Warehouse and Garbage Collection Intensional Value Ware- 
house and Garbage Collection were implemented by Lei Tao in |Tao04] . After integra- 



tion, his contributions became to look like as shown in Figure |68j The IValueHouse 
and its extension IVWInterf ace are the ones used by the Executor to communicate 
to a concrete implementation of a warehouse, allowing adding/changing warehouse 
implementations easily without affecting the Executor. All the exception handling 
is based on the GEEException. 



4.2.3.4 RIPE Design 



The class diagram describing RIPE is in Figure 69 The RIPE class represents a 
facade to the rest of the RIPE modules. It is semi-implemented, as many things are 
not clear on this side of the project yet. The only part of RIPE that was advanced 
well so far by Yimin Ding in jDin04j is the Data-Flow-Graph (DFG) editor, which 
is not implemented in Java. The DFGEditor Java class is meant to be main Java 
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RIPE 

from RIPE) 




IVWInspector 

(from Runtime System) 



^IVWlnspectorO 



Controller 

Ijrom Ru nti meSystem) 



^ControllerQ 



DFGEditor 

(from DFGEditor) 




TextualEditor 

(from editors) 















WebEditor 

(from WebEditor) 



♦WebEditorO 



Figure 69: RIPE Design. 

program acting like a bridge between Java and the LEFTY language, but did not get 
implemented yet. The rest of the modules are planned stubs. 



4.2.3.5 Data Flow Graphs Integration 



The integration of Yimin Ding's |Din04] DFG-related work is presented in Figure 70 
The DFGAnalyzer was augmented to implement the ICompiler interface as it fol- 
lows the same structure as the rest of our compilers, which compiles a Lucid code 
from DFG. The DFGException class, a subclass of GIPCException has been created 
to indicate an error situation in the DFG processing. DFGAnalyzer's SimpleNode 
was updated to inherit from GIPC . intesional . SimpleNode due to vast functional- 
ity overlap. The two analyzer and generator modules have been placed under the 
GIPC. DFG. DFGAnalyzer and GIPC. DFG. DFGGenerator packages. 
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^genASTO 
^genPointQ 
^gendimO 
^gencodeO 
^gennairiGQ 
^prtASTO 
^geriTableQ 
^geriTablecloseQ 
^linkNodeO 
^ndwhereO 
%indrootO 
^genASTO 
^genPointQ 
^gendimQ 
^gencodeQ 
^gennameQ 
^prtASTO 
^geriTableQ 
^genLCQ 



DFGTranCodeGGnerator 

(from DFOeaneratoO 



^DFGTranCodeGeneratorO 
^getSnameO 
^getPnameQ 
^drawWhereO 
^drawAssignQ 
^drawAssignLableQ 
^drawVarQ 
^drawDIMO 
^drawTerm20 
^drawTermQ 

^drawRankgroupO 
^drawDFGO 
^drawDFGcloseO 
^geriTableQ 
^geriTablecloseO 
^drawStarlQ 
^drawOperatorO 
♦drawDFGO 
^geriTableQ 
^generateDFGQ 
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-^sliowtreeO 
-^reepassQ 
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(from ore Analyzer) 



DFGCodeGenerator 

(from DFGGenerjtor) 



^DFGCodeGeneratorO 
^getSnameO 
^getPnameQ 
^drawWhereO 
^drawAssignQ 
^drawAssignLabieQ 
^drawVarQ 
^drawDIMO 
^drawTerm20 
^drawTermQ 
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^drawDFGO 

^drawDFGcloseO 
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^drawStartO 

^drawOperatorO 

^drawDFGO 
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DFGAnalyzer 

(from DFGAnalyzeO 



^strFilename : String 



^DPGAnaiyzerO 
♦initO 
^parseQ 
^compiieQ 

^g et Ab st ra ct S y nt a xTre e Q 
^runQ 

^getLastE>iceptionO 

^setSourcGCodeStreamO 

^compileQ 

^mainO 



DFGSubtree 

(from DF(JGeneraior) 

^SBT: Hashtable 
^nterm : int 
^used : int 



^DFGSubtreeO 

♦accVarO 

^ndVarO 

♦listKeysO 

^IsUsedO 

^reqNQ 

^getPnameO 

^getSnameO 



-current 

± 



DFGSubtreeContents 

(from DF&Generator) 



T^nterm : int 
'^used : int 



♦DFGSubtteeContents 
♦DFGSubtteeContents 



-o 



ICompiler 

(from <p\FC) 



Figure 70: DFG Integration Design. 
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4.3 Summary 



This chapter presented most of the development effort went into integration, design, 
and implementation of JLucid, Objective Lucid, and GICF. User interfaces (both web 
and command line) has been outlined. Regression Test Suite has been introduced. 
The follow up chapter presents a variety of testing approaches went into the GIPSY 
to prove successful integration of the old and implementation of new modules. 

To summarize. Objective Lucid, as opposed to GLU |JD96t IJDA97] and JLucid, 
provides access to the object members and is real object-oriented hybrid language. 
While JLucid may indirectly manipulate objects through pseudo-free functions, the 
actual objects are still a "black box" to it. 

The GICF and IPLCF gave an ability for an easier integration of intensional and 
imperative languages in the GIPSY. The below are the steps one needs to perform to 
add a new compiler to the GIPSY: 

• create a package where the language compiler will reside (usually under 
imperative/LANGUAGE or intensional/SIPL/LANGUAGE. 

• add a compiler class that extends either one of IntensionalCompiler, 
ImperativeCompiler, or implements one of their superinterfaces 

• the code segment and fully qualified class name should be added to either 
ElmperativeLanguages or EIntensionalLanguages 

• optionally implement a custom version of a preprocessor if it is a hybrid language 

• implement translation rules to GIPL if it is a SIPL if it is an intensional language 

• implement proper ST/CP generation for an imperative language according to 
that language's semantics and typing instructions 

• implement type mapping table upon the need if it is an imperative language 

The above might still sound complex, but it is much more easier and flexible than 
before. Additionally, some of the steps can be abstracted and simplified, but it is 
impossible to eliminate manual work altogether. 
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Chapter 5 



Testing 



This chapter addresses the testing aspect of this thesis for the following two main 
reasons: integration and refactoring of a variety of the GIPSY modules including 
GICF and the development and operation of the two new Lucid dialects developed 
in this work, namely JLucid and Objective Lucid. Notice, this testing is far from 
comprehensive and does not include testing of the execution performance of any of 
the programs and many compilation aspects are still to be resolved as of this writing 
(and be resolved in the final version). This is, however, a starting point of setting up 
the GIPSY testing infrastructure for the projects to come to do mandatory systematic 
tests, which are now a necessity given the size of the system, a centralized source tree, 
and the number of subprojects developed simultaneously. 

5.1 Regression Testing 
5.1.1 Introduction 

The regression testing is a comprehensive set of tests for the implementation and 
integration of the GIPSY modules. They test most of the operations and capabihties 
of the GIPSY. The test cases primarily are various intensional programs (hybrid 
or not) that exercise the main modules, such as GIPC and GEE as well as their 
submodules with the major focus on GIPC. 
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5.1.2 Regression Testing Suite 

The regression tests can be run against already pre-compiled gipsy . j ar, or by using 
a temporary installation within the source tree using the Regression application. 
Next, there are a "sequential" and "parallel" modes to run the tests. In the sequential 
mode tests run in strict sequence, whereas in the parallel mode multiple threads are 
started to run groups of tests in parallel. 



5.1.2.1 Unit Testing with JUnit 

The core of the Regression application is based on the JUnit framework introduced in 



Section 2.6.1.3 Regression represents a TestSuite, that contains ParallelTestCase 
and SequentialTestCase, a subclasses of TestCase. Both types of tests are cus- 
tomizable based on the options supplied to the Regression application (see Sec- 



tion 4.2.1.6). JUnit helps to tell us what errors happened and in which modules and 



the reason of the failures dynamically at run-time. 



5.1.2.2 Unit Testing with dif f 

It becomes cumbersome to use JUnit for all possible cases, in a large system, where 
often we are generally interested in the output behaviour changes only. Here the Unix 
tool dif f helps us. A collection of hand-checked outputs are said to be "expected", 
one ore more file for each test case. Then, when the next time the test is run, a 
current directory is created with the current outputs, and the current and expected 
output directories are compared with the dif f to show the differences in the output 
produced by the modules. This is all achieved by the regression script. 

5.1.2.3 Tests 

The actual test cases in the form of GIPL, Indexical Lucid, Objective Lucid, JLucid, 
and GIPSY programs, are located under the corresponding src/tests/* directories 
in the source tree in the form of * . ipl files. These comprise most of the examples 
presented earlier in this work as well as developed in |Paq99| , |Ren02j . |Wu02j . and 
|Lu04j . The regression tests for the DFG generation ( |Din04j ) . Intensional Value 
Warehouse and Garbage Collector |Tao04j and Demand Migration System (DMS) 
|VP05j are not present as of this implementation. 
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5.2 Portability Testing 



GIPSY has been tested and is known as expected (regression tests pass) to run 
on Red Hat Linux 9, Fedora Core 2, Mac OS X, Solaris 9, Windows 
98SE/2000/XP systems under JDK 1.4 and 1.5. The corresponding hardware ar- 
chitectures were Intel or Intel-compatible processors (Pentium II, III, and IV with 233 
MHz to 1.4 GHz) and G3 and G4 processors from Apple and IBM. For the WebEditor 
interface. Tomcat 5 on Mac OS X were tested, but it is believed to run on other 
platforms the Jakarta Tomcat runs on. 
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5.3 Solving Problems 



This section is targeting some common problems of synchronization in parallel and 
distributed environment and how they are solved using the GIPSY system relieving 
the programmer from the need of explicitly synchronize the objects. They also illus- 
trate the use of arrays and embedded Java, and Java objects. These programs are 
among many other test cases from the Regression Tests Suite. 



5.3.1 Prefix Sum 



pseudocode (for thread 'j') 

'shared' a 'future' 'int' 'array' [L.logP, 1..P] := undefined; 
'private' sum 'int' := j, 
hop 'int' := 1; 



'do' level = 1, logP > 

'if j <= P - hop > a[level, j] := sum 'fi' 

'if j > hop > sum +:= a [level, j - hop] 'fi' 

hop := 2 * hop 



Figure 71: Pseudocode of a thread j for the Prefix Sum Problem. 



/* 

* PREFIX SUM in GIPL-style JLucid program. 

* Numbers are from 1 to 8. 

* S[I] will contain prefix sum for number 'i' 
*/ 



#JLUCID 



// Array of prefix sums 

S[8] ®d 8 

where 

dimension d; 



S[I] = if (#d = 0) 
then 1 

else (2 * S[I] - 1) Sd (#d - 1) 

fi; 

// Index the array varies within. 

I ai 8 

where 

dimension i; 

I = if(#i = 0) 1 else (1 - 1) @d (#i - 1); 

end; 

end; 



Figure 72: The Prefix Sum Problem in JLucid in GIPL Style. 
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The pseudocode of for a thread j is in Figure 71 |Pro03aj . The Figure 72 shows 



the program translated into Lucid. The Figure 73 shows the program translated into 
Indexical Lucid for numbers from 1 to 8. Below is an equivalent implementation of 
the problem (targeting only TLP) in Java; compare the program's line count and 
complexity to that of JLucid: 



// Modified from Dr. Probst's Cyclic. Java 

public class PrefixSum 

{ 

public static final int P = 8; // number of workers 

public static final int logP =3; // number of rows in logP x P matrix 
// For permutation of workers 

private static int [] col = {3, 6, 5, 7, 4, 2, 1, 0}; 

// These two mimic a 2D array of future variables 

public static int [] [] a = new int [logP] [P] ; 

public static Semaphore [] [] futures = new Semaphore [logP] [P] ; 

// The resulting sums are to be placed here, 
public static int[] sums = new int[P]; 

public static void main(String [] argv) 
{ 

Worker w[] = new Worker [P] ; 



for(int j = 0; j < futures . length; j++ ) 

forCint k = 0; k < f utures [j] . length; k++) 
futures [j] [k] = new Semaphore(O) ; 

forCint j = 0; j < P; 
{ 

w[col[j]] = new Worker (col [j] + 1); 
w[col[j]] . start () ; 

} 



forCint j = 0; j < P; 
{ 

try 
{ 

w[j] . joinO ; 

} 

catch(InterruptedException e) 



forCint j = 0; j < P; 

System, out .println C'Prefix Sum of " + Cj + 1) + " = " + sums[j]); 
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System. out. print In ("System terminates normally."); 

} 

} 

class Semaphore 
{ 

private int value; 

Semaphore (int value 1) 
{ 

value = value 1 ; 

} 

public synchronized void WaitO 
{ 

try 
{ 

while (value <= 0) 
{ 

waitO ; 

} 

value — ; 

} 

catch (InterruptedException e) 

{ 

} 

} 

public synchronized void Signal () 
{ 

++value ; 
notif yO ; 

} 

} 

class Worker extends Thread 
{ 

private int j ; 
private int sum; 
private int hop = 1; 

public Worker(int col) 
{ 

sum = j = col; 

} 

public void run() 
{ 

System. out. print In ("Worker " + j + " begins execution."); 
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yieldO; 

for(int level = 0; level < Pref ixSum. logP; level++) 
{ 

if(j <= PrefixSum.P - hop) 
{ 

System . out . print In 
( 

"Worker " + j + 

" defines a[" + level + "," + (j-1) +"]." 

); 

Pref ixSum. a [level] [j - 1] = sum; 
PrefixSum. futures [level] [j - l].Signal(); 

} 

if(j > hop) 
{ 

Pref ixSum. futures [level] [j - 1 - hop].Wait(); 

System . out . pr intln 
( 

"Worker " + j + 

" uses a[" + level + "," + (j - 1 - hop) + "]." 

); 

sum += Pref ixSum. a [level] [j - 1 - hop]; 

} 

hop = 2 * hop; 

} 

PrefixSum. sums [j - 1] = sum; 

System. out. println ("Worker " + j + " terminates."); 

} 

} 
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/* 

* PREFIX SUM in Indexical Lucid-style JLucid 
*/ 

#JLUCID 

S[8] Sd 8 
where 

dimension d; 

S[I] = 1 fby.d (2 * S[I] - 1); 

I Si 8 
where 

dimension i; 

1=1 fby.i (I - 1); 

end; 

end; 



Figure 73: The Prefix Sum Problem in JLucid in Indexical Lucid Style. 
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5.3.2 Dining Philosophers 



Below is a JLucid implementation of the Dining Philosophers problem |Dij65 Dij71 



IGinQO] . We have arrays of 8 philosophers and 8 forks, each represented as integers. 
A philosopher is either thinking (1) or eating (2); likewise for forks, taken or not. 
A philosopher may eat when they have exactly two forks, not less, if the forks are 
available. If none available, the philosopher waits (implicit, guaranteed by the GEE). 
The special variable I serves as an intensional index for our arrays. 



/** 

* Dining Philosophers Problem 

* in JLucid 
* 

* Oauthor Serguei Mokhov, mokhovScs . concordia. ca 

* aversion $Revision: 1 . 10 $ $Date: 2005/03/02 02:57:31 $ 
*/ 



#f uncdecl 



int getlninitalRandomState () ; 

boolean chew (int) ; 

boolean brainstorinldea(int) ; 



#JLUCID 



/* 

* Assume 8 philosophers and two states. 

* States: 2 - eating, 1 - thinking 

* Forks are either available or not; hence, 2 states as well. 
*/ 

PHILOSOPHERS [8] Sstates 2 
where 

dimension states; 



// Initialize all forks 
FORKS [8] aavailability 2 
where 

dimension availability; 



FQRKS[I] = getlninitalRandomState () ; 



I ad 8 
where 

dimension d; 

1 = 1 fby.d (I - 1) ; 

end; 

end; 
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I* 

* Run the actual algorithm. 

* NOTE: In this Implementation the computation 

* never terminates (normally). It is an infinite loop. 
*/ 

PHILOSOPHERS [I] = 

if(#states == 1) then 
eat (I) Ostates 2 

eat (I) = 

getForks(I) && chew (I); 

getForks(I) = g(l, r) 
where 

1 = FORK [I] aavailability 1; 
r = FORK [I] aavailability 1; 

end; 

else 

think(I) Sstates 1 

think (I) = 

putForks(I) kk brainstormldea(I) ; 

putForks(I) = p(l, r) 
where 

1 = FORK [I] aavailability 2; 
r = FORK [I] aavailability 2; 

end; 

fi; 

I ad 8 
where 

dimension d; 

1=1 fby.d (I - 1); 

end; 

end; 
#JAVA 

int getlninitalRandomStateO 
{ 

// Either 1 or 2 

return new RandomO .nextlnt(2) + 1; 

} 

boolean chew(int i) 
{ 

try 
{ 

System. out. printlnC'Philo " + i + " is chewing smth tasty now."); 
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sleepCnew RandomO .nextlntCi * 1200)); 

System . out .printlnC'Philo " + i + " finished chewing."); 
return true; 

} 

catchdnterruptedException e) 
{ 

return false; 

} 



boolean brainstormldeaCint i) 
{ 

try 
{ 

System. out .printlnC'Philo " + i + " is heavily thinking now."); 
sleepCnew RandomO .next Int(i * 1200)); 

System. out .println("Philo " + i + " finished thinking."); 
return true; 

} 

catchdnterruptedException e) 
{ 

return false; 

} 
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5.3.3 Fast Fourier Transform 



This is an example on how one would compute Fast Fourier Transform (FFT) in the 
GIPSY for an array of double values. This is straightforward in Lucid because it's 
deterministic with plenty of parallelism. A JLucid program implementing FFT is in 



Section 5.3.3.1 The algorithm is based on the Java algorithm implemented in MARF 



[MCSNOSj IPre93l IBerOS] . a code fragment of which is in Section 5.3.3.2 originally 
written by Stephen Sinclair. The JLucid version omits the imaginary part of the 
transform, but it would not be hard to add it. 



5.3.3.1 Fast Fourier Transform in JLucid. 



/* 

* FFT implementation in JLucid. 

* Serguei Mokhov 

* $Id: fft.ipl.v 1.2 2005/08/13 01:37:23 mokhov Exp $ 
*/ 

#f uncdecl 

double sin(double) ; 
double pi () ; 

#JAVA 

double sin(double pdValue) 
i 

return Math. sin(pdValue) ; 

} 

double pi() 
{ 

return Math. PI; 

} 

#JLUCID 
A 

where 

// A is an array of 9 FFT values with a 
// normal FFT applied to the array below. 

A = fft([l, 2, 3, 4, 6, 7, 8, 9], 9, 1); 

f f t (inputValues , length, sign) = fftValues 
where 

fftValues = apply (length, reverse(length, inputValues), sign); 
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applyClen, coeffs, direction) = coeffs 9.s (N - 1) 
where 

dimension s; 



N = 2 * len; 

nmax = (2 fby.s istep) upon(iimiax < N) ; 

coeffs [J / 2] = coeffs [I / 2] - tempr; 
coeffs [I / 2] = coeffs [I / 2] + tempr; 

where 

istep = mmax fby.s (istep) * 2; 

M S.m mmax 
where 

dimension m; 

M = (0 fby.m (M + 2)) upon (M < mmax); 

tempr = wr * coeffs [J / 2] - wi * coeffs [J / 2]; 

J = I + mmax; 

wr = 1.0 fby.m ((wtemp = wr) * wpr - wi * wpi + wr) ; 
wi = 0.0 fby.m (wi * wpr + wtemp * wpi + wi) ; 

where 

dimension i; 

I = (M fby.i (I + istep)) upon (I < H); 
theta = (direction * 2 * piO) / mmax; 
wtemp = sin(0.5 * theta); 
wpr = -2.0 * wtemp * wtemp; 
wpi = sin (theta) ; 

end; 

end; 

end; 

end; 

// Binary reversion 
reversed, vals) = out S.i length 
where 

dimension i; 

out[t] = vals[#.i] @ (#.i + 1) a. bit maxbits (length) ; 
where 

dimension bit; 

t = fby.bit ((t * 2) I (n & 1)); 
n = #i fby.bit (n / 2) ; 

end; 

end; 
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// Determine max number of bits 
maxbits(len) = (mblts - 1) ®.m 16 
where 

dimension m; 

mblts = ( fby.m (mblts + 1) ) upon (mblts < 16 fe& n != 0) ; 
n = len fby.m (n / 2); 

end; 

end; 

end; 
// EOF 



5.3.3.2 Fcist Fourier Transform code fragment in Java from MARF. 



/** 

* <p>FFT algorithm, translated from "Numerical Recipes In C++" that 

* implements the Fast Fourier Transform, which performs a discrete Fourier transform 

* in O(n*log(n)) .</p> 
* 

* Sparam InputReal InputReal is real part of input array 

* Sparam Inputlmag Inputlmag Is Imaginary part of input array 

* Oparam OutputReal OutputReal is real part of output array 

* Sparam Outputlmag Outputlmag is Imaginary part of output array 

* Sparam direction Direction is 1 for normal FFT, -1 for inverse FFT 

* Sthrows MathException if the sizes or direction are wrong 
*/ 

public static final void doFFT 
( 

final double [] InputReal, 
double [] Inputlmag, 
double [] OutputReal , 
double [] Outputlmag, 
Int direction 

) 

throws MathException 
{ 

// Ensure input length is a power of two 
int length = InputReal . length ; 

if ((length < 1) I ((length & (length - 1)) != 0)) 

throw new MathException("Length of input (" + length + ") is not a power of 2."); 

if ((direction != 1) M (direction != -1)) 

throw new MathException("Bad direction specified. Should be 1 or -1."); 

if (OutputReal . length < InputReal . length) 

throw new MathException ("Output length (" + OutputReal . length + ") < Input length (" + InputReal . length + 
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// Determine max number of bits 
int maxbits, n = length; 



for(maxbits = 0; maxbits < 16; maxbits++) 
{ 

if(n == 0) break; 
n /= 2; 

} 

maxbits -= 1; 

// Binary reversion & interlace result real/imaginary 
int i, t, bit; 

forCi = 0; i < length; i++) 
{ 

t = 0; 
n = i; 

for (bit = 0; bit < maxbits; bit++) 
{ 

t = (t * 2) I (n & 1); 
n /= 2; 

} 

OutputReal [t] = InputReal [i] ; 
OutputImag[t] = Inputlmag[i] ; 

} 

// put it all back together (Danielson-Lanczos butterfly) 

int mmax = 2, istep, j, m; // counters 

double theta, wtemp, wpr, wr, wpi, wi, tempr, tempi; // trigonometric recurrences 

n = length * 2; 

while (mmax < n) 
{ 

istep = mmax * 2; 

theta = (direction * 2 * Math. PI) / mmax; 

wtemp = Math.sin(0.5 * theta); 

wpr = -2.0 * wtemp * wtemp; 

wpi = Math. sin (theta) ; 

wr = 1.0; 

wi =0.0; 

for(m = 0; m < mmax; m += 2) 
{ 

ford = m; i < n; i += istep) 
{ 

j = i + mmax; 

tempr = wr * OutputReal [j / 2] - wi * Output Imag[j / 2]; 
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tempi = wr * Output Imag[j / 2] + wi * OutputReal[j / 2]; 

OutputRealLj / 2] = OutputReal[i / 2] - tempr; 
OutputImag[j / 2] = OutputImag[i / 2] - tempi; 

OutputReal[i / 2] += tempr; 
OutputImag[i / 2] += tempi; 

} 

wr = (wtemp = wr) * wpr - wi * wpi + wr; 
wi = wi * wpr + wtemp * wpi + wi; 

} 

mmax = istep; 

} 

} 
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5.3.4 Moving Car 



A less contrived example of an Objective Lucid program is presented in Figure [74 
This is an example where a Car object changes with time. Eliminating S, and ignoring 
the print call, we have have: 



C O.time 15 where 

C = CarO fby.time C. move (#. time) 

Using the definition of f by gives: 



C O.time 15 

= (CarO fby.time C. move (#. time)) O.time 15 

= if 15 <= then Car() else (C.move(#.time)) @.time (15 - 1) 
= C. move (14) 



Our intention is that f by will give the sequence: 



CarO Car. move (1) Car. move (2) . . . Car. move (15) 



This will work as follows. When one generates a demand for C. move (15) it's not 
satisfied until C.move(14) is until C.move(13) is ... until C.move(l) is until CarO, 
so it recurses back and finally the CarO object instance gets constructed, and then 
the demands fiow from 1 to 15 and the instance already exists. 

The car also does not accelerate indefinitely. It moves until it has enough fuel, 
else it returns the car object with its members unmodified. The drop of speed is also 
in place when fuel is depleted. 

To further illustrate this idea let's take the existing example of a simpler problem 



of natural numbers presented in Figure 22 and convert it into Objective Lucid as in 



Figure 76 First, we will present the eduction tree of the natural numbers problem 



see Figure 75, a corrected version of the one produced by Paquet in |Paq99| ) and 



then transmute it into the eduction tree of the execution of the equivalent Objective 



Lucid propgram, as shown in Figure [77| The program in Figure [76] exhibits the same 
properties as the Car example, so the eduction tree will be similar but will take more 
space. The important aspect here is to illustrate the difference between demands 
for STs and their lazy execution (which is italisized, e.g. N.inc()); thus, the actual 
invocation of a ST method happens at a later time after the demand is made so we 
avoid not having called constructor prior execution of an instance method. In the 
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#typedecl 
Car; 



#JAVA 

public class Car 
{ 



public 


int X 


= 0; 


public 


float 


speed; 


public 


float 


speeddrop; 


public 


float 


fuel; 


public 


float 


f ueldrainrate ; 


public 


CarO 




// 


Assume initially car 



speed = 100.0; fuel = 40.5; 

f ueldrainrate = 0.018; speeddrop = 0.1; 

} 

// Move by a number of steps assuming constant speed 
// and decelerate when ran out of fuel, 
public Car move (int steps) 
{ 

if (fuel > 0) 
{ 

fuel -= f ueldrainrate * speed * steps; 
X += steps; 

} 

else if (speed > 0) 
{ 

X += steps; 

speed -= speeddrop * steps; 

} 

return this; 



public void printCarStateO 
{ 

System . out . println 
( 

"Speed: " + speed + ", fuel: " + fuel + 

", drain: " + f ueldrainrate + ", x: " + x + 

" , speeddrop : " + speeddrop 

); 

} 

#OBJECTIVELUCID 

(C a. time 15) .printCarStateO 
where 

C = CarO fby.time S; 
S = C.move(#time) ; 

end; 



Figure 74: Objective Lucid example of a Car object that changes in time. 
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eduction trees the normal arrows correspond to demands made for expressions and 
the bullet arrows correspond to the result of evaluation of the demands, which are 
also bold and italic. In the Objective Lucid eduction tree object instance is denoted 
as ClassName:MemberName:value and the {d:X} presents the context of evaluation. 
The result of evaluation of the Objective Lucid variant is said to be true because, as 
previously defined, void methods are mapped to return true and the last expression 
bit that is evaluated here is the print () method call of the instance of a Nat32 class, 
which returns void. 
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Figure 



75: Eduction Tree forTtJie Natural Numbers Problem. 



#typedecl 
Nat42; 

#JAVA 

class Nat42 
{ 

private int n; 

public Nat42() 
{ 

n = 42; 

} 

public Nat42 incO 
{ 

n++; 

return this; 

} 

public void print 
{ 

System. out. printlnC'n = " + n) ; 

} 

} 

#OBJECTIVELUCID 

(N O.d 2) .print [d] () 
where 

dimension d; 

N = Nat42[d]() fby.d N.inc[d](); 

end 



Figure 76: The Natural Numbers Problem in Objective Lucid. 
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Figure 77: Eduction Tree for the NatujgL^Numbers Problem in Objective Lucid. 



5.3.5 Game of Life 

The Game of Life [Gar 70] would make a good benchmark for the GIPL. Life takes 
place on a 2D grid and evolves in time, so it's a 3D problem. The value of a cell at 
time T + 1 depends on the value of the cell and its 8 neighbours at time T. Thus, there 
is a high branching factor and the IVW will get plenty of exercise. Peter Grogono 
wrote a version in Haskell, which is functional and lazy but is not concurrent and 
does not have an IVW. The author of this work made a version in Indexical Lucid. 



In Figure 78 is the top-level function. The Game of Life program is included in the 
test suite as a good elaborate test case, but this work does not address any of the 
performance and efficiency issues related to the execution and wareshousing, so no 
measurements have been done two compare the efficiency of the program with and 
without the warehouse nor with the Haskell program. 

life = evolve T initial (conway life) where 
initial = F(\i -> 

if val Y i == M <= val X i && val X i < 5 then 1 else 0) 

conway v = F(\i -> 

let neighbours v = 

ev V (n i) + ev v (ne i) + ev v (e i) + ev v (se i) + 

ev V (s i) + ev v (sw i) + ev v (w i) + ev v (nw i) in 

b2i (neighbours v == 3 I I ev v i == 1 && neighbours v == 2)) 

evolve d s e = F(\i -> 

if val d i == then ev s i else ev e (prev d i)) 

b2i b = if b then 1 else 

n i = F(. . .) 

Figure 78: The Life in Haskell. 

Explanations: 

• evolve{d, u,v) allows a value to evolve in the dimension d. The first value of 
the stream is given by u and subsequent values by v. 

• initial{d) defines the initial configuration (five ones in the row 0, zeroes every- 
where else in the matrix 5-by-5). 

• conway{d, v) computes the successor of state v. The functions n, ne, e, se, s, 
sw, w, and nw are "navigators" that find values of neighbours. 

• b2i{d) converts a Boolean to integer to decide the new value of an entity. 
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#INDEXICALLUCID 

life = evolved, initial(T), conwayClife, T)) 
where 

dimension T; 

evolve (d, u, v) = u fby.d v; 

initial(d) = 

if (Y == && <= X && X < 5) then 1 else 
where 

X = fby.d X + 1; 

Y = fby.d Y + 1; 

end; 

conwayCd, v) = b2i (neighbours ==311 (v == 1 && neighbours == 2)) 
where 

neighbours = n(d) + ne(d) + e(d) + se(d) + s(d) + sw(d) + w(d) + nw(d) ; 
where 



n(d) 


= V « 


!.(d - 


5), 


ne(d) 


= V « 


!.(d - 


4), 


e(d) 


= V « 


!. (d + 


1), 


se(d) 


= V « 


!. (d + 


6), 


s(d) 


= V « 


1. (d + 


5), 


sw(d) 


= V « 


l.(d + 4), 


w(d) 


= V « 


!.(d - 


1), 


nw(d) 


= V « 


!.(d - 


6), 



end; 

b2i(b) = if(b) then 1 else 0; 

end; 

end; 



Figure 79: The Life in Indexical Lucid. 
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5.4 Summary 

There were many tests developed and exercised for the GIPSY. This section attempted 
to show the reader the most representative ones and how the Regression Tests Suite 
works in the GIPSY for the most modules of GIPC and GEE and how JUnit is apphed 
to make it possible and maintainable. Now, every new module added to the GIPSY 
system will have to have a corresponding unit and/or regression test (or several tests) 
exercising most of the features of this module added. 
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Chapter 6 



Conclusion 



To conclude, it is believed GIPSY is well off the ground and is steadily getting ready 
for its first large public release to the research community. It is becoming a lot 
more usable not only by a small circle of GIPSY developers, but also by scientists 
and researchers from other research groups. Preliminary testing (see Chapter p« and 



results (Section 6.1 ) give confidence in the success of an important step for the GIPSY 
in the are of flexible hybrid intensional-imperative programming. To summarize, the 
newly introduced features for the innovative intensional research platform GIPSY are 
a valuable asset allowing us to release GIPSY to the masses and a new release will be 



made at the SourceForge.net at http : / /sf . net/pro j e cts/sf gipsy circa the end of 
December 2005 - January 2006. 



6.1 Results 

6.1.1 Experiments 

The experiments conducted on the GIPSY research platform were primarily design, 
development, and testing of hybrid programming paradigms by fusing together inten- 
sional and imperative languages. For test experiments please refer to Chapter |5j 

6.1.2 Interpretation of Results 

After extensive testing of the design and implementation of ideas presented in Chap- 
ter [3] we can see an enhanced, more flexible GIPSY system taking off the ground. 
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Most of regression tests pass for the developed sample programs with known errors 
and failures. 

6.2 Discussions and Limitations 

6.2.1 Lack of Hybrid Intensional-Imperative Semantics Proofs 

The semantics for the GIPSY Type System was not defined and the one of JLucid 
and Objective Lucid was not formally proven to be correct. 

6.2.2 Genuine Imperative Compilers 

The most serious limitation of the current implementation of the hybrid paradigm is 
that there are no genuine imperative GIPSY compilers. The Java wrapper compiler 
classes merely resort to the external tools from the library of enumerated tools. This 
makes overall error checking and reporting cumbersome. Additionally, this slows 
down the compilation process. 

6.2.3 Cross-Language Data Type Mapping 

When implementing other imperative language compilers than Java, or a genuine 
compiler for Java, a special mapping has to be explicitly established in the form of 
TypeMap. We can avoid this for C/C++ with the JNI |Ste05j . but not for other 
popular languages. 

6.2.4 Dimension Index Overflow 

While this limitation is not directly related to the main topics of this thesis, it has 
to be mentioned. In the current implementation of the dimension type in all Lucid 
variants is done as a simple Java integer, and as such, is finite. Thus, incorrectly 
written Lucid programs or programs that may require high dimension values may 
overflow the dimension index rendering execution of the program incorrect. This 
limitation is not handled by the GEE nor constrained in the operational semantics of 
Lucid. 
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6.2.5 Hybrid-DFG Integration 

This thesis does not address placement, rendering, and integration of the hybrid AST 
nodes into DFGs. 

6.2.6 Dealing With Side Effects and Abrupt Termination 

As of this implementation, GEE has very hmited control over what's happening inside 
the STs in terms side effects, exceptions, non-termination, etc. in the Java (or other 
imperative language) code causing it to exit prematurely or to hang. Likewise, we 
cannot do warehousing of non-immutable STs due to the side effects, i.e. when the 
same arguments are given to an ST may yield a different result at different invocations. 
This is serious aspect, which is related to the development of any future semantics of 
the hybrid programming languages and deserves a separate publication. 

6.2.7 Imperative Function Overloading 

It is an error to write the following: 
#funcdecl 
int f oo(int) ; 
int f 00 (float) ; 



but it shouldn't be. This is an error in the sense that only the last declaration is 
retained due to the way function identifiers are handled, so no function overloading 
at this moment is officially supported. The issue of dealing with the semantics of a 
type system in which this is possible, especially if we support multiple imperative PLs, 
where each may have potentially its own type system or even paradigm is complex. 
However, this feature is nice to have and some practical aspects can be implemented, 
which will be a research topic on its own. 

6.2.8 Cross-Imperative Language Calls 

Normally, an ST written say in #JAVA cannot call another ST in say #C. This limitation 
is that only the intensional part can make calls to the imperative functions. This 



163 



eliminates the need to keep the type mappings between all possible combinations of 
the imperative languages and semantics associated with this. 

However, depending on the language, procedures written in the same language 
can possibly communicate by calling each other. E.g. in Java, defining free members 
and passing state between free functions is possible as nothing is done to prevent this. 

#JAVA 

int i; 

int fooO { 

return i + 1; 

} 

int barO { 
i++; 

return f oo() ; 

J 

This is based on the knowledge about the internal implementation i.e. the "int 
i;" bit will also be wrapped in the class, so it'd be legal to have it from the Java's 
point of view; however, is considered to be a kludge and non-portable feature. To 
be on the safer side, the STs like that should be written assuming no knowledge of 
internal state for communication is available. 

6.2.9 Security 

JLucid, Objective Lucid, and GICF opened up doors for very fiexible use of external 
languages and resources as a part of intensional computation. Unfortunately, there 
are security considerations to deal with when embedding a vulnerable unsigned code 
from possibly untrusted remote location and then propagate it to all the workers par- 
ticipating in computation can result resulting either gaining some unwanted privileges 
to the attackers or DDoS. 
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Chapter 7 
Future Work 



The future work to take on will focus in the following areas to either address the 



limitations outlined in Section 6.2 or to introduce new features, not necessarily all 



related to the topics of this thesis. 

• Integration of the Demand Migration System (DMS) [VP05j . 

• Formal semantic verification from Indexical Lucid through Objective Lucid. 

• Placement of hybrid nodes into DFGs. 

• Security. 

• Trial C compiler with JNL 

• Fully Explore Array Properties. 

• Genuine imperative compilers in GICF. 

• Introduction functional language compilers. 

• Visualization and control of communication patterns and load balancing. 

• Target Host Compilation. 

• Java wrapper for the DFG Editor of Yimin Ding. 
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7.1 Formal Verification of Semantic Rules and the 
GIPSY Type System 



One needs to formally conduct verification proofs of the semantic rules from Indexical 
Lucid to Objective Lucid in PVS or Isabelle, so this project can be undertaken in 
the near future and the work on it has already began. Specifically, a relation to the 
semantic of objects and Java's operational semantics has to be made. Likewise, the 
semantics of the newly introduced GIPSY type system has to be formally defined. 

7.2 Dealing with Data Flow Graphs in Hybrid Pro- 
gramming 

This thesis did not deal with the way on how to augment DFGAnalyzer and DFGGenerator 
to support hybrid GIPSY programs. This can be addressed by adding an unexpand- 
able imperative DFG node to the graph. To make it more useful, i.e. expandable and 
so it's possible to generate the GIPSY code off it or reverse it, would require having 



the genuine compilers as in Section |7.6| for imperative languages, which is far from 
trivial. 



7.3 Security 

Security is a substantial concern in distributed computing. The great flexibility pro- 
vided by embedded Java in JLucid (and later in Objective Lucid) can be misused 
and be a source of security breaches or DDoS attacks (e.g., due to explicit oversyn- 
chronization using Java's synchronization primitives explicitly). Thus, the follow-up 
work in this direction would include malicious code detection in embedding and dis- 
tributing as well as explicit synchronization points so that there are no deadlocks and 
DDoS potential is reduced. This concern touches the compiler (GIPC), the Generator- 
Worker architecture, the GIPSY Server, and the GIPSY Screen Saver components of 
the GIPSY system. 
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7.4 Implementation of the C Compiler in GICF 



An methodology of implementing a C compiler, and therefore, C CPs and STs has 
been devised, but never implemented, so in the future a C compiler will be imple- 
mented part of GICF with the JNI [5te05] . 

7.5 Fully Explore Array Properties 

The arrays in JLucid, Objective Lucid, and their generalization in GICF requries 
further exploration and formalization and mapping of the GIPSY arrays to their 
native equivalents. 

7.6 Genuine Imperative and Functional Language 
Compilers 

Future work in this area is to focus on writing our genuine compilers for the mentioned 
imperative languages and extending support for more imperative and functional lan- 
guages (e.g. LISP, Scheme, or Haskell) and make it as much automated as possible. 

7.7 Visualization and Control of Communication 
Patterns and Load Balancing 

It is proposed to have a "3D editor" within RIPE's DemandMonitor that will render 
in 3D space the current communication patterns of a GIPSY program in execution 
or replay it back and allow the user visually to redistribute demands if they go off 
balance between workers. A kind of virtual 3D remote control with a mini expert 
system, an input from which can be used to teach the planning, caching, and load- 
balancing algorithms to perform efficiently next time a similar GIPSY application is 
run. 
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7.8 Target Host Compilation 



This has to do with enabhng the GEE to dehver the ST source code around and 
compile it on the target host instead of sending a pre-compiled version of the STs. 
This is an experimental feature can be useful and dangerous and requires a lot of 
research. 



7.9 The GIPSY Screen Saver 



This is a sample implementation of a worker, outlined in Section 3. 3.3.4 would rep- 
resent an application for a PC as a way to contribute to a GIPSY program execution. 
Three sample implementations of screen saver workers exist one for Windows, one for 
Linux and one for MacOS X. 



7.10 The GIPSY Server 

A so-called "GIPSY server" will be implemented to be able to serve intensional or oth- 
erwise requests primarily through the HTTP protocol, thus acting like a mini-GIPSY 
intensional web server. It would accept request from remote clients via HTTP or local 
clients via command line and be the starting point of computation (an intensional 
computation resource) available to all those who have no resources to set up GIPSY. 
This is not duplicate any of the DMS [VP05j nor it is a part of RIPE, as it is primarily 
non-interactive and runs on the background. 
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Appendix A 



Definitions and Abbreviations 



A.l Abbreviations 

AST - Abstract Syntax Tree 
COM - Component Object Model 

CORBA - Common Object Requester Broker Architecture 
CLP - Cluster-Level Parallelism 



CP - Communication Procedure, Section 3.3.3.2 

CVS - Concurrent Versions System 

DOOM - Distributed COM 

DDoS - Distributed Denial of Service (attack). 

FFT - Fast Fourier Transform 

FTP - File Transfer Protocol 



DPR - Demand Propagation Resource, Section 2.5.4.1 |RG05al [PWOSj 
GEE - General Eduction Engine 



GEER - GEE Resources, Section 4.1.1.14 



GIPC - General Intensional Program Compiler, Figure 10, |RG05at IPWOSj 
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GIPL - General Intensional Programming Language, |Paq99 fRGOSat fPWOSj 



GIPSY - General Intensional Programming System, |RG05at IPWOSj 
GLU - Granular Lucid, pD96l IJDA971 |Paq99| 
HTTP - Hyper- Text Transfer Protocol 



IDP - Intensional Demand Propagator, Section 3.3.3.4 |RG05at IPWOS] 
IDS - Intensional Data-dependency Structure 
IP - Intensional Programming 

IPL - Intensional Programming Language (e.g. GIPL, GLU, Lucid, Indexical 
Lucid, JLucid, Tensor Lucid, Objective Lucid, Onyx |Gro04] ) 



IVW - Intensional Value Warehouse, Section |3.3.3.4| |R(;n5at IPWn5] 

JDK - Java Developer's Kit 

JNI - Java Native Interface 

JRE - Java Runtime Environment 

JSSE - Java Secure Socket Extension 

MARF - Modular Audio Recognition Framework |MCSN05j 

MPI - Message Passing Interface 

NCP - Native Communication Procedure 

NST - Native Sequential Thread 

NUMA - Non-Uniform Memory Access 

PVM - Parallel Virtual Memory System 



RFE - Ripe Function Executor, Section [3X34 |R(;n5al IPWOS] 
RMI - Remote Method Invocation 
RPC - Remote Procedure Call 
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SIPL - Specific IPL (e.g. Indexical Lucid, JLucid, Tensor Lucid, Objective 
Lucid, Onyx) 

SLP - Stream-Level Parallelism 



ST - Sequential Thread, Section |3.3.3.1 
TLP - Thread-Level Parallelism 
TTS - Time To Solution 
UMA - Uniform Memory Access 
URI - Unified Resource Indentifier 
URL - Unified Resource Locatior 
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Appendix B 

Sequential Thread and 
Communication Procedure 
Interfaces 



In this section the actual definitions of the CP and ST interfaces, an example of a 
generated wrapper class and a Worker are presented. 

B.l Sequential Thread Interface 



See Figure 80 



B.2 Communication Procedure Interface 



See Figure 81 



B.3 Generated Sequential Thread Wrapper Class 

This is a more complete version of the generated wrapper class for the code in Fig- 
ure I 
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package gipsy . interfaces ; 

import Java. io . Serializable ; 
import Java. lang. reflect. Method; 

/** 

* <p>Sequential Thread represents a piece work to be done. 

* Has to extend Serializable for RMI, CORBA, COM+, Jini to work. 

* Rvmnable needed to run it in a separate thread. </p> 
* 

* $Id: ISequentialThread. java.v 1.13 2005/09/12 01:24:38 mokhov Exp $ 
* 

* aversion $Revision: 1.13 $ 

* Sauthor Serguei Mokhov, mokhov@cs.concordia.ca 

* asince Inception 
*/ 

public interface ISequentialThread 
extends Runnable, Serializable 
{ 

/** 

* Work-piece to be done. 

* Sreturn WorkResult container 
*/ 

public WorkResult workO; 

public WorkResult getWorkResult ; 
public void setMethod (Method poSTMethod) ; 

} 

// EOF 



Figure 80: Sequential Thread Interface. 



182 



package gipsy . interfaces ; 
import gipsy . lang . GIPSYType ; 
import java.io.Serializable; 

/** 

* <p>CoiiimunicationProcedure represents the means of delivery of sequential threads. </p> 

* $Id: ICommunicationProcedure. java.v 1.11 2005/10/11 08:34:11 mokhov Exp $ 

* aversion $Revision: 1.11 $ 

* Oauthor Serguei Mokhov, mokhovScs . concordia. ca 

* Osince Inception 

* Osee gipsy . interfaces . SequentialThread 
*/ 

public interface ICommunicationProcedure 

extends Serializable 

{ 

public GIPSYType getRetumType () ; 

public GIPSYType getParamTypeCf inal int piParamNumber) ; 

public GIPSYType [] getParamlypesO ; 

public void setReturnType (GIPSYType polype); 

public void setParamType (final int piParamNumber, GIPSYType poType) ; 

public void setParamTypes (GIPSYType [] paoTypes) ; 

public GIPSYType getParamType (String pstrLexeme) ; 

public GIPSYType getParamType (String pstrLexeme, String pstrlD) ; 

public int getParamListSizeO ; 

/** 

* Perform any initialization actions required. 

* areturn status object of the result of send operation. 

* athrows CommunicationException in case of error 
*/ 

public CommunicationStatus initO 
throws CommunicationException; 
/** 

* Open a connection; whatever that means for a given protocol. 

* Sreturn status object of the result of send operation. 

* @throws CommunicationException in case of error 
*/ 

public CommunicationStatus openO 
throws CommunicationException; 
/** 

* Close a connection; whatever that means for a given protocol. 

* Qreturn status object of the result of send operation. 

* @throws CommunicationException in case of error 
*/ 

public CommunicationStatus close () 
throws CommunicationException; 
/** 

* Defines the means of sending data. Should be overridden by 

* a concrete implementation, such as JINI, COM, CORBA, etc. 

* areturn status object of the result of send operation. 

* athrows CommunicationException in case of error 
*/ 

public CommunicationStatus sendO 
throws CommunicationException; 
/** 

* Defines the means of receiving data. Should be overridden by 

* a concrete implementation, such as JINI, COM, CORBA, etc. 

* Sreturn status object of the result of receive operation. 

* athrows CommunicationException in case of error 
*/ 

public CommunicationStatus receiveO 
throws CommunicationException; 



Figure 81: Communication Procedure Interface. 
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import java.util .Hashtable; 
import java.util .Vector ; 



public class <f ilename>_<machine_naine>_<timestamp> 
implements gipsy . interfaces . ISequentialThread 
{ 

private OriginalSourceCodeInf o oOriginalSourceCodeInf o ; 
/** 

* Inner class with original source code information 
*/ 

public class OriginalSourceCodeInf o 
{ 

/** 

* For debugging / monitoring; generated statically 
*/ 

private String strOriginalSource = 
"int getN(int piDimension) " + 
"{" + 

" if (piDimension <= 0)" + 
" return get42() ; " + 

else" + 

" return getN (piDimension - 1) + 1;" + 

"}" + 

III! + 

"int get42()" + 
"{" + 

" return 42 ; " + 
"}"; 

/** 

* Mapping to original source code position for error reporting 
*/ 

private Hashtable oLineNumbers = new Hashtable (); 
/** 

* Body is filled in by the preprocessor statically 
*/ 

public OriginalSourceCodeInf o() 
{ 

Vector int_getN_int_piDimension = new VectorO; 

// Start line and Length in lines 

int_getN_int_piDimension . add(new IntegerO)) ; 
int_getW_ int .piDimension. add (new Integer (7)) ; 

thi s . oLineNumbers . put 
( 

"int getN (int piDimension)", 
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int_getN_ int _p ID Imens ion 

); 

Vector int_get42 = new VectorO; 

int _get42 . add (new Integer(ll)); 
int_get42 . add(new Integer(4)); 

this . oLineNumbers . put 
( 

"int get42()", 
int_get42 

); 

} 

public Hashtable getLineNumber sHash ( ) 
{ 

return this . oLineNumbers ; 

} 

public int getLineNumberForFunction(String pstrFunctionSignature) 

{ 

} 

public int getFunctionSourceLength(String pstrFunctionSignature) 

{ 

} 

public String toStringO 

{ 

} 

} 

/** 

* Constructor 
*/ 

public <f ilename>_<machine_name>_<timestamp>() 
{ 

this . oOriginalSourceCodeInf o = new OriginalSourceCodeInf o() ; 

} 

public String toStringO 
{ 

return this . oOriginalSourceCodeInf o . toString ( ) ; 

} 

I* 

* Implementation of the SequentialThread interface 
*/ 

// Body generated by the compiler 
public void run() 
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{ 

Payload oPayload = new PayloadO ; 
oPayload.addC'd" , new Integer(42)) ; 

work (oPayload) ; 

} 

// Body generated by the compiler statically 
public WorkResult work (Payload poPayload) 
{ 

WorkResult oWorkresult = new WorkResult () ; 
oWorkresult . add (getN (poPayload . getVaueOf ( " d " ) ) ) ; 
return oWorkResult; 

} 

/* 

If 

* The below are generated off the source file nat2java.ipl 
If 

*/ 

public static int getN(int piDimension) 
{ 

if (piDimension <= 0) 
return get42(); 

else 

return getN (piDimension - 1) + 1; 

} 

public static int get42() 
{ 

return 42; 

} 
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B.4 Sample Worker's Implementation 



package gipsy . wrappers ; 

//import gipsy . interfaces . SequentialThread; 
import gipsy . interfaces . ICommunicationProcedure ; 
import gipsy .util .* ; 

import marf . util . BaseThread ; 

/** 

* Worker Class Definition 
* 

* IRevision: 1.11 $ by $Author: mokhov $ on $Date: 2004/11/06 00:50:09 $ 
* 

* Sversion $Revision: 1.11 $ 

* Sauthor Serguei Mokhov 
*/ 

public class Worker extends BaseThread 
{ 

/** 

* Aggregation of sequential threads. 
*/ 

private Thread [] aoSequentialThreads = null; 
/** 

* Set of available communication procedures for different protocols. 

*/ 

private ICommunicationProcedure [] aoCommuncationProcedures = null; 
/** 

* Default settings. 
*/ 

public WorkerO 
{ 

super ; 

} 

/** 

* Generate a demand. 

*/ 

public void demand () 

{ 

} 

/** 

* Receive a result on a demand. 
*/ 

public void receive 
{ 
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} 



/** 

* Perform computation. 

*/ 

public void workO throws GIPSYException 
{ 

try 
{ 

for(int i = 0; i < this. aoSequentialThreads. length; i++) 
this . aoSequentialThreads [i] . start ; 

} 

catch (NullPointerException e) 
{ 

throw new GIPSYException 
( 

"Worker TID=" + getTIDO + 

" did not have any sequential threads to work on." 

); 

} 

} 

/** 

* Stops worker thread. 
*/ 

public void stopWorkerO 

{ 

} 

/** 

* From Runnable interface, for TLP 
*/ 

public void run() 
{ 

try 
{ 

workO ; 

} 

catch (GIPSYException e) 
{ 

System. err. println(e) ; 

} 

} 

} 

// EOF 
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Appendix C 



Architectural Module Layout 



C.l GIPSY Java Packages and Directory Struc- 
ture 

Normally, a directory structure of a Java project corresponds to the package naming; 
thus, the packages are named and declared after the directories. By the means of 
Java packages, all the classes within the project and external applications "know" 
how to identify and import the classes they intend to use. A fully-qualified class 
name includes all the packages starting from the "root" (the top-level directory of 
the hierarchy) all the way up to the class itself, separated by a dot. The GIPSY Java 



packages breakdown as of this writing corresponds to the Figure |82 

The logical breakdown was performed in accordance with the original conceptual 
design primarily produced by Joey Paquet and further by Aihua Wu and Bo Lu, has 
been the primary source of the hierarchy plus any exceptions and extensions that 
various team members come up with or have been forced to during implementation 
were taken into account. 

The basic structure is as follows. The top root hierarchy is logically the gipsy 
package. The major non-utility packages under it, which come from the conceptual 
design, are GIPC, GEE, and RIPE. The major utility packages under gipsy that are 
not present in the conceptual design are: interfaces for most intermodule commu- 
nication; wrappers for object wrapping; storage for the serializable interface classes; 
util for most common exceptions and utility modules (e.g. fast linked list |Din04] ) : 
and tests for the Unit and Regression Testing Suites. 
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Figure 82: GIPSY Java Packages Hierarchy. 
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Under the GIPC package the major modules (to be discussed later in this chapter) 
include Preprocessing for general GIPSY program preprocessing, intensional and 
imperative language compilers and their necessary followers (GenericTranslator 
for the former and CommunicationProcedureGenerator and SequentialThreadGenerator 
for the latter) . Then the DFG package for Lucid-to-data-flow-graph and back genera- 
tion. 

The gee's main packages includes IDP for demand propagation and IVW for caching 
and garbage collection. 

Under RIPE we have interactive run-time editing and monitoring modules that 
include textual editor, DFG editor, and the web-based editor. 

C.2 GIPSY Modules Packaging 

gipsy's major and minor modules are packaged into a set of runnable .jar files and 
distributed with wrapper scripts to be either used as ordinary command line tools 
as a part of GIPSY Development Kit or the WebEditor web apphcation. Different 
. j ar files include a subset of all GIPSY modules depending on the need, e.g. GIPC 
includes GIPC-related classes plus GEE as we allow to optionally invoke GEE after 
successful compilation. RIPE, except itself, needs both GIPC and GEE, whereas 
GEE does not at all require presence of any other module. Thus, the GIPSY binary 
distribution is broken down into five major . j ar files (notice, that these files do not 
include any external libraries GIPSY references): 

• gipsy . j ar simply includes almost all of GIPSY. 

• gipc.jar should be used/distributed as a part of so-called "GIPSY Develop- 
ment Kit" if someone wants to be able to compile intensional programs and 
optionally run them. 

• gee. jar represents GIPSY's non-interactive run-time environment, the GEE. 
This can be distributed alone to the hosts that only wish to run pre-compiled 
GIPSY programs and have no development environment set up. 

• ripe, jar includes most of the interactive programming environment of the 
GIPSY along with GIPC and GEE. 
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Table 2: Correspondence of the GIPSY .jar files and the modules. 
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• Regression, jar includes the Regression Testing application plus all of GIPC 
and GEE as the most exercised modules for testing as of this writing. 

The Table |2] shows correspondence between the variety of modules and their con- 
tainment within a . j ar file. 
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Appendix D 



Grammar Generation Scripts for 
JLucid and Objective Lucid 

D.l jlucid.sh 



#! /bin/bash 

strDate='date' 

cat «GRAMMAR_TAIL 
/* 

* Generated by jlucid.sh on $strDate 
*/ 

/** 

* ©since $strDate 
*/ 

void embed #EMBED : {} 
{ 

//<EMBED> <LPAREN> url() E() ( <COMMA> E() )* <RPAREN> <SEMICOLON> 

<EMBED> <LPAREH> url() <COMMA> <STRING_LITERAL> ( <COMMA> E() )* <RPAREN> <SEMICOLON> 

} 

/** 

* 9since $strDate 
*/ 

void array #ARRAY : {} 
{ 

<LBRACKET> E() ( <COMMA> E() )* <RBRACKET> 

} 

/** 

* URL -> CHARACTER_LITERAL I STRING_LITERAL . 
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* Sslnce $strDate 

*/ 

void urlO #URL : 
{ 

Token oToken; 

} 
{ 

( 

oToken = <CHARACTER_LITERAL> 
I oToken = <STRING_LITERAL> 

) 
{ 

jjtThis.setlmageCoToken. image) ; 

} 

} 

// EOF 

GRAMMAR.TAIL 
# EOF 



D.2 JGIPL.sh 



#! /bin/bash 

cat . ./. ./GIPL/GIPL. jjt I \ 

# Filter out unneeded stuff 

grep -V '//EOF' I \ 

#grep -V 'import gipsy .GIPC. intensional . SimpleNode' I \ 

# Fix package 

sed ' s/intensional\ .GIPL/intensional\ .SIPL\ . JLucid/g' I \ 

# JLucid GIPL 

sed ' s/GIPL/ JGIPL/ ' I \ 

sed 's/\/\/{EXTEND-E}/\/\/{EXTEND-E}\n\t\t| embed ()/' I \ 
sed 's/\/\/{EXTEND-FACTOR}/\/\/{EXTEND-FACTOR}\n\t| array ()/' I \ 
sed 's/<WHERE: "where">/<WHERE: "where">\n\t I <EMBED: "embed">/g' \ 
> JGIPL. jjt 

./jlucid.sh » JGIPL. jjt 

# EOF 



D.3 JIndexicalLucid. sh 

#! /bin/bash 

cat . . / . . /SIPL/IndexicalLucid/IndexicalLucid. j jt I \ 
# Filter out unneeded stuff 



194 



grep -V V/ EOF' I \ 

#grep -V 'import gipsy .GIPC. intensional . SimpleNode' I \ 

# Fix package 

sed ' s/intensional\ . SIPL\ . IndexicalLucid/intensionalN . SIPL\. JLucid/g' I \ 

# JLucid Indexical 

sed 's/IndexicalLucid/JIndexicalLucid/' I \ 

sed ' s/\/\/{EXTEND-E}/\/\/{EXTEND-E}\n\t\t I embedO/' I \ 

sed 's/\/\/{EXTEND-FACTOR}/\/\/{EXTEND-FACTOR}\n\t| array ()/' I \ 

sed 's/<WHERE: "where">/<WHERE: "where">\n\t I <EMBED: "embed">/g' \ 

> JIndexicalLucid. j jt 

./jlucid.sh >> JIndexicalLucid. jjt 

# EOF 



D.4 ObjectiveGIPL.sh 



#! /bin/bash 

cat JGIPL.jjt I \ 

# Filter out uimeeded stuff 

grep -V '//EOF' I \ 

# Fix package 

sed 's/intensional\.SIPL\. JLucid/intensional\.SIPL\.ObjectiveLucid/g' I \ 

# DbjectiveLucid GIPL 

sed 's/JGIPL/ObjectiveGIPL/' I \ 

sed ' s/\/\/{EXTEND-El}/\/\/{EXTEND-El}\n\t\t\t I ( <DOT> ID() ) #OBJREF El()/' \ 
> ObjectiveGIPL. jjt 

# EOF 



D.5 ObjectivelndexicalLucid. sh 



#! /bin/bash 

cat JIndexicalLucid. jjt I \ 

# Filter out unneeded stuff 
grep -V '//EOF' I \ 

# Fix package 

sed ' s/intensional\ . SIPL\ . JLucid/ intensional\ . SIPL\ .ObjectiveLucid/g' I \ 

# ObjectiveLucid Indexical 

sed ' s/JIndexicalLucid/ObjectivelndexicalLucid/ ' I \ 

sed 's/\/\/{EXTEND-El}/\/\/{EXTEND-El}\n\t\t\t| ( <DOT> 1D() ) #OBJREF El()/' \ 
> ObjectivelndexicalLucid. jjt 

# EOF 
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